Recenlty, I was searching for newer USDA quads and came across the USDA site that has updated USDA quads for National Forest areas [ Link ]. This is very useful since the National Forest service road network changes fairly frequently and my USDA quads from Libremap (some of them) are old. I found out while navigating along non-existant roads :-)
The main problem however with the USDA site is it really is designed to prohibit you from downloading a large set of map data at one time, and forces you to download the data via small geographic segments. I wanted to download all the maps for the Western US all at once. Download managers didn’t work either.
So, I created a process to download the map data.
[ LINK ] — Here is a link to data I have already downloaded seperated into UTM Zones and States. This data is zipped with 7zip, and contains the .tif file, the .tfw world file, the .fdg file when available, the html metadata file and finally, the OziExplorer .map file. *will be added later
Here is the process I used in case it helps someone else get the data directly:
Requirements:
A. GNU Unix utilities (sed 3.02.80 or higher, wget, cp and cat) [ Link ]
B. WinHTTrack (website copier) [ Link ]
C. comb.bat (my batch file to create the file list)
D. uncmp.bat (my batch file to un-gzip, un-tar and then move the *.tar.gz’d files)
E. 7zip a file compression/decompression utility [ Link ]
F. Bulk Rename Command [ Link]
[ LINK ] — Here are all the above, my batch files, along with the binaries for the other utilities. The “run_apps” directory should be at the same level as the batch files so that the references work correctly to the binaries.
——————————————
The Basic process is as follows:
1. Use comb.bat to get html files from the USDA National Forest website
2. Use comb.bat to process the html files and generate a second list of URLS which point to the gzipped map data
3. Use WinHTTrack to download the tar.gz files and the html metadata files for the maps
4. Use uncmp.bat to process the html metadata files and ungzip and untar the map data into UTM Zone directories
5. Import the resulting UTM directories into OziExplorer
Use comb.bat to do the following:
1. Create a text file containing all the states to be retrieved. The text file should be named “states_2_get.txt”, and it should contain a 2 capital letter abbreviation for each state to be retrieved.
For example:
CO
AZ
CA
UT
2. Using wget and the list of states we copy the html pages locally from the National Forest website to process
3. using sed and findstr we process the html files and generate text files containing the links to the directories which contain the map files (in tar.gz format)
4. Then we move all the resulting text files into a newly created subdirectory [process] and concatonate those files and do some cleanup on the URLs. Then we modify the URLs to point to the actual file location.
5. This results in a file named WinHTTrack_File_List.txt which is a source file used by WinHTTrack to download a the tar.gz files from the National Forest Service website to your local computer.
——————————————
Use WinHTTrack to do the following:
1. Create a new project in WinHTTrack, select your destination drive (make sure its large)
a. in the Action drop down select “Get separated files”
b. Use the file WinHTTrack_File_List.txt as the “URL list (.txt) file for the source URL’s
c. Click on “Set options…”
d. in the Options dialog box, select the “Scan Rules” tab and use:
+*.html
+*.zip +*.tar +*.tgz +*.gz +*.rar +*.z +*.exe
e.
f. on the “Limits” tab set max mirroring depth=2, max external depth=2
g. on the “Flow Control” tab set Number of Connections=8 (this depends on your bandwidth
h. on the “Links” tab make sure to check the “Get non-HTML files related to a link” checkbox
i. you can leave the other items the way they are or adjust to your preference
j. Finish to start the download
K. Go do something else, this may take days to complete the download
2. When you want to update this, later on, if you keep the original tar.gz files, you can use WinHTTrack to get just the new files.
——————————————
Use uncmp.bat to do the following:
1. Well, now you have a huge bunch of files in a directory structure like this
Q:\UPDATED_NAT_FOREST\NationalForest\fsgeodata.fs.fed.us\rastergateway\softcopy
30093
30094
30095
31093
31094
31095
31108
etc…
Inside those directories are the *.tar.gz files
2. from the cmd line, run uncmp.bat, passing 3 arguments to the batch file:
uncmp.bat D:\source_path\with_wildcards\*\ Y:\destination\directory\ tar
– the first argument is the source path and it can contain wild cards this is
the path to the directory that contains the subdirectories with the *tar.gz files
– the second argument is the destination directory you want the files output into
– the last argument is the search string to identify the files (in our case tar)
3. Running this batch file will parse the html files for the UTM Zone, and create a
series of directories in the destination folder for each UTM zone. Then the tar.gz
files will be ungzipped, untarred into a proper UTM zone directory, and into a subdir
for collar or uncollared whichever is appropriate. Then the html files metadata files
which have been renamed to cooraspond to the actual map names are moved into those
same UTM Zone directories.
4. Import these UTM Zone directories into OziExplorer for you mapping pleasure.
**Note, the batch files have a lot of commenting in them so, please check there with any questions.
No related posts.
