Optimal method to convert raster from 32bit Float to 8bit Byte to reduce file size in QGIS 3 The Next CEO of Stack OverflowWhy does an 32-bit FGDB raster have a smaller file size than 8-bit?Extracting raster classe(s) and creating new raster that is SMALLER in size using QGIS?How to set all pixels with value <= 0 to “nodata” in DEM raster?Creating Maximum-Value Composite using QGIS?Polygonizing only pixel values of 1 from raster using QGIS?QGIS raster calculator error: cannot convert float infinity to integerWhy does an 32-bit FGDB raster have a smaller file size than 8-bit?Is is possible to have some 0-value pixels as NoData while others not in GeoTiff using GDAL?Translate (convert format): Set the size of the output fileVirtual Raster Mosaic - gdalbuildvrt - data gapsQGIS 3.2 - Poor conversion from high-resolution vector to rasterQGIS raster calculator not keeping each pixel with 1 byte?
How do I avoid eval and parse?
Is micro rebar a better way to reinforce concrete than rebar?
How did people program for Consoles with multiple CPUs?
Rotate a column
Should I tutor a student who I know has cheated on their homework?
How to start emacs in "nothing" mode (`fundamental-mode`)
Preparing Indesign booklet with .psd graphics for print
Do I need to enable Dev Hub in my PROD Org?
Is "for causing autism in X" grammatical?
Is it possible to search for a directory/file combination?
Why do remote companies require working in the US?
What does "Its cash flow is deeply negative" mean?
How do scammers retract money, while you can’t?
Anatomically Correct Strange Women In Ponds Distributing Swords
Would this house-rule that treats advantage as a +1 to the roll instead (and disadvantage as -1) and allows them to stack be balanced?
Is it ever safe to open a suspicious html file (e.g. email attachment)?
To not tell, not take, and not want
Novel about a guy who is possessed by the divine essence and the world ends?
How to invert MapIndexed on a ragged structure? How to construct a tree from rules?
Which tube will fit a -(700 x 25c) wheel?
Is there a difference between "Fahrstuhl" and "Aufzug"
How do I go from 300 unfinished/half written blog posts, to published posts?
How do we know the LHC results are robust?
Why am I allowed to create multiple unique pointers from a single object?
Optimal method to convert raster from 32bit Float to 8bit Byte to reduce file size in QGIS 3
The Next CEO of Stack OverflowWhy does an 32-bit FGDB raster have a smaller file size than 8-bit?Extracting raster classe(s) and creating new raster that is SMALLER in size using QGIS?How to set all pixels with value <= 0 to “nodata” in DEM raster?Creating Maximum-Value Composite using QGIS?Polygonizing only pixel values of 1 from raster using QGIS?QGIS raster calculator error: cannot convert float infinity to integerWhy does an 32-bit FGDB raster have a smaller file size than 8-bit?Is is possible to have some 0-value pixels as NoData while others not in GeoTiff using GDAL?Translate (convert format): Set the size of the output fileVirtual Raster Mosaic - gdalbuildvrt - data gapsQGIS 3.2 - Poor conversion from high-resolution vector to rasterQGIS raster calculator not keeping each pixel with 1 byte?
I have a raster file (.tif) that contains integer values from 1 to 31. I have realized that the values are stored as Float32 with NoData Value=-3.39999999999999996e+38 (see raster information below). I think that the optimal method to store the data is as unsigned 8 bit integer (QGIS Byte) as there are no decimal values and the maximum value is below 255. I have used the Translate (convert format) tool available from GDAL to change the data type from Float32 to Byte (following these instructions). However, I have run into two issues:
- The resulting file is BIGGER than my original file (from 31MB to 268MB). I noticed that the original file had
COMPRESSION=LZWand tried to use various compression formats underProfilebut I was not successful in reducing file size while maintaining data quality (e.g. usingJPEGwith 100 resulted in some values being +/-1 of the values they should be and some random additional pixels) - I had difficulty setting the
NoDatavalue. It seems that0worked in my case but I may have files with 0-values that are meaningful and I wouldn't know how to convert the multidecimal NoData value from the Float to an integer value that is not confused with actual zero values in the dataset.
Or are compression techniques more optimal with Float32 than with Byte so file sizes are smaller even though it is storing it as a higher-memory file? Perhaps it is also relevant to note that there are many more pixels with NA values than pixels with values.
Size is 21768, 12920
Pixel Size = (250.000000000000000,-250.000000000000000)
Metadata:
AREA_OR_POINT=Area
Image Structure Metadata:
COMPRESSION=LZW
INTERLEAVE=BAND
Band 1 Block=21768x1 Type=Float32, ColorInterp=Gray
Min=1.000 Max=31.000
Minimum=1.000, Maximum=31.000, Mean=nan, StdDev=nan
NoData Value=-3.39999999999999996e+38
UPDATE:
According to this related post, the raster size is computed based on the following formula:
The size of a raster is just the product of bit-depth/8, bands, rows,
and columns plus header metadata (statistics, etc.)
32/8* 1* 21768* 12920* =1124970240 Bytes ~1124MB
8/8* 1* 21768* 12920* =281242560 Bytes ~281MB
That gives me roughly the size Windows indicates for my 8-bit raster (281MB~268MB). What about the treatment of NA values? Does an NA pixel take up as much space as a pixel with a value? Is the 32-bit compression algorithm more successful in not using digital space for the cells without data than the 8-bit compression algorithm?
qgis raster qgis-3 format
asked yesterday
user3386170user3386170
648522
add a comment |
I have a raster file (.tif) that contains integer values from 1 to 31. I have realized that the values are stored as Float32 with NoData Value=-3.39999999999999996e+38 (see raster information below). I think that the optimal method to store the data is as unsigned 8 bit integer (QGIS Byte) as there are no decimal values and the maximum value is below 255. I have used the Translate (convert format) tool available from GDAL to change the data type from Float32 to Byte (following these instructions). However, I have run into two issues:
- The resulting file is BIGGER than my original file (from 31MB to 268MB). I noticed that the original file had
COMPRESSION=LZWand tried to use various compression formats underProfilebut I was not successful in reducing file size while maintaining data quality (e.g. usingJPEGwith 100 resulted in some values being +/-1 of the values they should be and some random additional pixels) - I had difficulty setting the
NoDatavalue. It seems that0worked in my case but I may have files with 0-values that are meaningful and I wouldn't know how to convert the multidecimal NoData value from the Float to an integer value that is not confused with actual zero values in the dataset.
Or are compression techniques more optimal with Float32 than with Byte so file sizes are smaller even though it is storing it as a higher-memory file? Perhaps it is also relevant to note that there are many more pixels with NA values than pixels with values.
Size is 21768, 12920
Pixel Size = (250.000000000000000,-250.000000000000000)
Metadata:
AREA_OR_POINT=Area
Image Structure Metadata:
COMPRESSION=LZW
INTERLEAVE=BAND
Band 1 Block=21768x1 Type=Float32, ColorInterp=Gray
Min=1.000 Max=31.000
Minimum=1.000, Maximum=31.000, Mean=nan, StdDev=nan
NoData Value=-3.39999999999999996e+38
UPDATE:
According to this related post, the raster size is computed based on the following formula:
The size of a raster is just the product of bit-depth/8, bands, rows,
and columns plus header metadata (statistics, etc.)
32/8* 1* 21768* 12920* =1124970240 Bytes ~1124MB
8/8* 1* 21768* 12920* =281242560 Bytes ~281MB
That gives me roughly the size Windows indicates for my 8-bit raster (281MB~268MB). What about the treatment of NA values? Does an NA pixel take up as much space as a pixel with a value? Is the 32-bit compression algorithm more successful in not using digital space for the cells without data than the 8-bit compression algorithm?
qgis raster qgis-3 format
asked yesterday
user3386170user3386170
648522
If the size of the data increases when you convert to BYTE, you are probably not going to be able to make your data much smaller than it originally is. Do you need the image to be an image or could you use a different format? For example, if the most important thing is data size rather than visualization, you could read in the data as an array and then write it to a hdf file. You wouldn't be able to visualize this, but you could still access the values for analysis
– Nathan Thomas
17 hours ago
I am using QGIS to play around and investigate data values across my study area -- does that imply "visualization"? Sorry for my naiveté.
– user3386170
12 hours ago
add a comment |
I have a raster file (.tif) that contains integer values from 1 to 31. I have realized that the values are stored as Float32 with NoData Value=-3.39999999999999996e+38 (see raster information below). I think that the optimal method to store the data is as unsigned 8 bit integer (QGIS Byte) as there are no decimal values and the maximum value is below 255. I have used the Translate (convert format) tool available from GDAL to change the data type from Float32 to Byte (following these instructions). However, I have run into two issues:
- The resulting file is BIGGER than my original file (from 31MB to 268MB). I noticed that the original file had
COMPRESSION=LZWand tried to use various compression formats underProfilebut I was not successful in reducing file size while maintaining data quality (e.g. usingJPEGwith 100 resulted in some values being +/-1 of the values they should be and some random additional pixels) - I had difficulty setting the
NoDatavalue. It seems that0worked in my case but I may have files with 0-values that are meaningful and I wouldn't know how to convert the multidecimal NoData value from the Float to an integer value that is not confused with actual zero values in the dataset.
Or are compression techniques more optimal with Float32 than with Byte so file sizes are smaller even though it is storing it as a higher-memory file? Perhaps it is also relevant to note that there are many more pixels with NA values than pixels with values.
Size is 21768, 12920
Pixel Size = (250.000000000000000,-250.000000000000000)
Metadata:
AREA_OR_POINT=Area
Image Structure Metadata:
COMPRESSION=LZW
INTERLEAVE=BAND
Band 1 Block=21768x1 Type=Float32, ColorInterp=Gray
Min=1.000 Max=31.000
Minimum=1.000, Maximum=31.000, Mean=nan, StdDev=nan
NoData Value=-3.39999999999999996e+38
UPDATE:
According to this related post, the raster size is computed based on the following formula:
The size of a raster is just the product of bit-depth/8, bands, rows,
and columns plus header metadata (statistics, etc.)
32/8* 1* 21768* 12920* =1124970240 Bytes ~1124MB
8/8* 1* 21768* 12920* =281242560 Bytes ~281MB
That gives me roughly the size Windows indicates for my 8-bit raster (281MB~268MB). What about the treatment of NA values? Does an NA pixel take up as much space as a pixel with a value? Is the 32-bit compression algorithm more successful in not using digital space for the cells without data than the 8-bit compression algorithm?
qgis raster qgis-3 format
asked yesterday
user3386170user3386170
648522
I have a raster file (.tif) that contains integer values from 1 to 31. I have realized that the values are stored as Float32 with NoData Value=-3.39999999999999996e+38 (see raster information below). I think that the optimal method to store the data is as unsigned 8 bit integer (QGIS Byte) as there are no decimal values and the maximum value is below 255. I have used the Translate (convert format) tool available from GDAL to change the data type from Float32 to Byte (following these instructions). However, I have run into two issues:
- The resulting file is BIGGER than my original file (from 31MB to 268MB). I noticed that the original file had
COMPRESSION=LZWand tried to use various compression formats underProfilebut I was not successful in reducing file size while maintaining data quality (e.g. usingJPEGwith 100 resulted in some values being +/-1 of the values they should be and some random additional pixels) - I had difficulty setting the
NoDatavalue. It seems that0worked in my case but I may have files with 0-values that are meaningful and I wouldn't know how to convert the multidecimal NoData value from the Float to an integer value that is not confused with actual zero values in the dataset.
Or are compression techniques more optimal with Float32 than with Byte so file sizes are smaller even though it is storing it as a higher-memory file? Perhaps it is also relevant to note that there are many more pixels with NA values than pixels with values.
Size is 21768, 12920
Pixel Size = (250.000000000000000,-250.000000000000000)
Metadata:
AREA_OR_POINT=Area
Image Structure Metadata:
COMPRESSION=LZW
INTERLEAVE=BAND
Band 1 Block=21768x1 Type=Float32, ColorInterp=Gray
Min=1.000 Max=31.000
Minimum=1.000, Maximum=31.000, Mean=nan, StdDev=nan
NoData Value=-3.39999999999999996e+38
UPDATE:
According to this related post, the raster size is computed based on the following formula:
The size of a raster is just the product of bit-depth/8, bands, rows,
and columns plus header metadata (statistics, etc.)
32/8* 1* 21768* 12920* =1124970240 Bytes ~1124MB
8/8* 1* 21768* 12920* =281242560 Bytes ~281MB
That gives me roughly the size Windows indicates for my 8-bit raster (281MB~268MB). What about the treatment of NA values? Does an NA pixel take up as much space as a pixel with a value? Is the 32-bit compression algorithm more successful in not using digital space for the cells without data than the 8-bit compression algorithm?
qgis raster qgis-3 format
qgis raster qgis-3 format
asked yesterday
user3386170user3386170
648522
asked yesterday
user3386170user3386170
648522
edited 12 hours ago
user3386170
asked yesterday
user3386170user3386170
648522
asked yesterday
user3386170user3386170
648522
asked yesterday
user3386170user3386170
648522
648522
If the size of the data increases when you convert to BYTE, you are probably not going to be able to make your data much smaller than it originally is. Do you need the image to be an image or could you use a different format? For example, if the most important thing is data size rather than visualization, you could read in the data as an array and then write it to a hdf file. You wouldn't be able to visualize this, but you could still access the values for analysis
– Nathan Thomas
17 hours ago
I am using QGIS to play around and investigate data values across my study area -- does that imply "visualization"? Sorry for my naiveté.
– user3386170
12 hours ago
add a comment |
If the size of the data increases when you convert to BYTE, you are probably not going to be able to make your data much smaller than it originally is. Do you need the image to be an image or could you use a different format? For example, if the most important thing is data size rather than visualization, you could read in the data as an array and then write it to a hdf file. You wouldn't be able to visualize this, but you could still access the values for analysis
– Nathan Thomas
17 hours ago
I am using QGIS to play around and investigate data values across my study area -- does that imply "visualization"? Sorry for my naiveté.
– user3386170
12 hours ago
If the size of the data increases when you convert to BYTE, you are probably not going to be able to make your data much smaller than it originally is. Do you need the image to be an image or could you use a different format? For example, if the most important thing is data size rather than visualization, you could read in the data as an array and then write it to a hdf file. You wouldn't be able to visualize this, but you could still access the values for analysis
– Nathan Thomas
17 hours ago
If the size of the data increases when you convert to BYTE, you are probably not going to be able to make your data much smaller than it originally is. Do you need the image to be an image or could you use a different format? For example, if the most important thing is data size rather than visualization, you could read in the data as an array and then write it to a hdf file. You wouldn't be able to visualize this, but you could still access the values for analysis
– Nathan Thomas
17 hours ago
I am using QGIS to play around and investigate data values across my study area -- does that imply "visualization"? Sorry for my naiveté.
– user3386170
12 hours ago
I am using QGIS to play around and investigate data values across my study area -- does that imply "visualization"? Sorry for my naiveté.
– user3386170
12 hours ago
add a comment |
0
active
oldest
votes
Your Answer
StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "79"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);
else
createEditor();
);
function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: false,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: null,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);
);
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fgis.stackexchange.com%2fquestions%2f317031%2foptimal-method-to-convert-raster-from-32bit-float-to-8bit-byte-to-reduce-file-si%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
0
active
oldest
votes
0
active
oldest
votes
active
oldest
votes
active
oldest
votes
Thanks for contributing an answer to Geographic Information Systems Stack Exchange!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fgis.stackexchange.com%2fquestions%2f317031%2foptimal-method-to-convert-raster-from-32bit-float-to-8bit-byte-to-reduce-file-si%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
If the size of the data increases when you convert to BYTE, you are probably not going to be able to make your data much smaller than it originally is. Do you need the image to be an image or could you use a different format? For example, if the most important thing is data size rather than visualization, you could read in the data as an array and then write it to a hdf file. You wouldn't be able to visualize this, but you could still access the values for analysis
– Nathan Thomas
17 hours ago
I am using QGIS to play around and investigate data values across my study area -- does that imply "visualization"? Sorry for my naiveté.
– user3386170
12 hours ago