Hfrhyu

Venezuelan girlfriend wants to travel the USA to be with me. What is the process?

What mechanic is there to disable a threat instead of killing it?

How do conventional missiles fly?

How much of data wrangling is a data scientist's job?

Forming a German sentence with/without the verb at the end

Does the Idaho Potato Commission associate potato skins with healthy eating?

Forgetting the musical notes while performing in concert

Determining Impedance With An Antenna Analyzer

What does the expression "A Mann!" means

How to properly check if the given string is empty in a POSIX shell script?

Can mass be shunted off into hyperspace, but the matter remains?

How does having to sign to support someone for elections fit with having a secret ballot?

Why would the Red Woman birth a shadow if she worshipped the Lord of the Light?

Which is the best way to check return result?

What is the most common color to indicate the input-field is disabled?

What method can I use to design a dungeon difficult enough that the PCs can't make it through without killing them?

Why do bosons tend to occupy the same state?

Assassin's bullet with mercury

Plagiarism or not?

Is there an expression that means doing something right before you will need it rather than doing it in case you might need it?

Would Slavery Reparations be considered Bills of Attainder and hence Illegal?

Calculating entropy change: reversible vs irreversible process

How do I deal with an unproductive colleague in a small company?

Madden-Julian Oscillation (MJO) - How to interpret the index?

Arcpy multiprocessing arcpy.Select_analysis

Why Can't ArcPy Search the Newly Created FeatureArcGIS Copy Features tool extremely slow when exporting to ArcSDE?Multiprocessing issues with ArcPySetting ArcPy environment variables with Python multiprocessing?Arcpy and multiprocessing not workingBatch intersect and multiprocessing with arcpyCombining python (parallel) multiprocessing toolbox with arcpyAccessing differing geodatabase schemas from arcpyarcpy and multiprocessing: switch workflow to shapefiles to avoid schemalocksMultiprocessing GALayerToRasters

1

I have the following script:

import arcpy
from arcpy import env 
import datetime
import os 


env.workspace = r"Q:\SDEDirect Connection to Contours.sde"
dataFC = arcpy.ListFeatureClasses("","","Contours")
outpath = r'C:UsersTESTDesktopSDE Conversion ProjectTEST.gdb'
query = "objectid > 0 AND objectid <= 200000 "

for fc in dataFC: 
 if fc == 'Contours':
 output = outpath + os.sep + fc
 arcpy.Select_analysis(fc, output+'200k', query)

Currently it selects records that have objectid between 0 and 200,000 and creates a feature class called Contours 200k in the TEST gdb.

What I want to be able to do is make a multiprocessing script that utilises one core to calculated between object id 0 and 200k, another that calculates betweek 200,001 and 400k and so on until the whole feature class is copied over as separate feature sets to the TEST gdb.

How would I go about doing that?
I'm trying to do this since the bottleneck of copying from the sde is the network connection and testing with separate files where I had the ranges set manually improved the overall speed a lot.

arcpy select parallel-processing

share|improve this question

edited 2 days ago

PolyGeo♦

53.9k1781245

asked 2 days ago

AndyAndy

141

New contributor

Andy is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.

• 
  
  
  

  
  1
  

  
  
  
  
  
  

  
  
  ArcGIS desktop is a single thread application, you can use subprocess.Popen to call multiple scripts but you will run into a self locking situation as only one process can write to a geodatabase at a time, you could create a GDB in your os.environ.get('TEMP') for each process and write there then use append or merge to join the multiple databases together though I would think that this would be no quicker in the long run... select by location on the other hand could be split up like this to gain better performance.
  
  – Michael Stimson
  2 days ago
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  I wrote a short blog providing a template over on geonet, suggest you read that to understand how to implement multiprocessing in arcpy. There are also many other related threads on geonet you should explore.
  
  – Hornbydd
  2 days ago
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  This has the feel of an XY Problem. The best way to improve contour line performance is to intersect with a coarse fishnet and dissolve on fishnet_fid (or FID plus elevation). This reduces the extent of individual features to the point that the spatial index is useful and reduces the feature count so that it is performant. Multi-threading is not generally a solution to a networking bottleneck.
  
  – Vince
  2 days ago
  

  
  
  
  

add a comment | 

1

I have the following script:

import arcpy
from arcpy import env 
import datetime
import os 


env.workspace = r"Q:\SDEDirect Connection to Contours.sde"
dataFC = arcpy.ListFeatureClasses("","","Contours")
outpath = r'C:UsersTESTDesktopSDE Conversion ProjectTEST.gdb'
query = "objectid > 0 AND objectid <= 200000 "

for fc in dataFC: 
 if fc == 'Contours':
 output = outpath + os.sep + fc
 arcpy.Select_analysis(fc, output+'200k', query)

Currently it selects records that have objectid between 0 and 200,000 and creates a feature class called Contours 200k in the TEST gdb.

What I want to be able to do is make a multiprocessing script that utilises one core to calculated between object id 0 and 200k, another that calculates betweek 200,001 and 400k and so on until the whole feature class is copied over as separate feature sets to the TEST gdb.

How would I go about doing that?
I'm trying to do this since the bottleneck of copying from the sde is the network connection and testing with separate files where I had the ranges set manually improved the overall speed a lot.

arcpy select parallel-processing

share|improve this question

edited 2 days ago

PolyGeo♦

53.9k1781245

asked 2 days ago

AndyAndy

141

New contributor

Andy is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.

• 
  
  
  

  
  1
  

  
  
  
  
  
  

  
  
  ArcGIS desktop is a single thread application, you can use subprocess.Popen to call multiple scripts but you will run into a self locking situation as only one process can write to a geodatabase at a time, you could create a GDB in your os.environ.get('TEMP') for each process and write there then use append or merge to join the multiple databases together though I would think that this would be no quicker in the long run... select by location on the other hand could be split up like this to gain better performance.
  
  – Michael Stimson
  2 days ago
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  I wrote a short blog providing a template over on geonet, suggest you read that to understand how to implement multiprocessing in arcpy. There are also many other related threads on geonet you should explore.
  
  – Hornbydd
  2 days ago
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  This has the feel of an XY Problem. The best way to improve contour line performance is to intersect with a coarse fishnet and dissolve on fishnet_fid (or FID plus elevation). This reduces the extent of individual features to the point that the spatial index is useful and reduces the feature count so that it is performant. Multi-threading is not generally a solution to a networking bottleneck.
  
  – Vince
  2 days ago
  

  
  
  
  

add a comment | 

I have the following script:

import arcpy
from arcpy import env 
import datetime
import os 


env.workspace = r"Q:\SDEDirect Connection to Contours.sde"
dataFC = arcpy.ListFeatureClasses("","","Contours")
outpath = r'C:UsersTESTDesktopSDE Conversion ProjectTEST.gdb'
query = "objectid > 0 AND objectid <= 200000 "

for fc in dataFC: 
 if fc == 'Contours':
 output = outpath + os.sep + fc
 arcpy.Select_analysis(fc, output+'200k', query)

Currently it selects records that have objectid between 0 and 200,000 and creates a feature class called Contours 200k in the TEST gdb.

What I want to be able to do is make a multiprocessing script that utilises one core to calculated between object id 0 and 200k, another that calculates betweek 200,001 and 400k and so on until the whole feature class is copied over as separate feature sets to the TEST gdb.

How would I go about doing that?
I'm trying to do this since the bottleneck of copying from the sde is the network connection and testing with separate files where I had the ranges set manually improved the overall speed a lot.

arcpy select parallel-processing

share|improve this question

edited 2 days ago

PolyGeo♦

53.9k1781245

asked 2 days ago

AndyAndy

141

New contributor

Andy is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.

import arcpy
from arcpy import env 
import datetime
import os 


env.workspace = r"Q:\SDEDirect Connection to Contours.sde"
dataFC = arcpy.ListFeatureClasses("","","Contours")
outpath = r'C:UsersTESTDesktopSDE Conversion ProjectTEST.gdb'
query = "objectid > 0 AND objectid <= 200000 "

for fc in dataFC: 
 if fc == 'Contours':
 output = outpath + os.sep + fc
 arcpy.Select_analysis(fc, output+'200k', query)

share|improve this question

edited 2 days ago

PolyGeo♦

53.9k1781245

asked 2 days ago

AndyAndy

141

New contributor

Andy is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.

share|improve this question

edited 2 days ago

PolyGeo♦

53.9k1781245

asked 2 days ago

AndyAndy

141

New contributor

Andy is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.

edited 2 days ago

PolyGeo♦

53.9k1781245

edited 2 days ago

PolyGeo♦

53.9k1781245

asked 2 days ago

AndyAndy

141

New contributor

Andy is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.

asked 2 days ago

AndyAndy

141