The dataset used in this example notebook is from the Nakadake Sanroku Kiln Site Center in Japan. The data set is provided by Shinoto et al. under the CC-BY-4.0 license: DOI
Creating filter pipelines
[1]:
import afwizard
This Jupyter notebook explains the workflow of creating a ground point filtering pipeline from scratch. This is an advanced workflow for users that want to define their own filtering workflows. For basic use, try choosing a pre-configured, community-contributed pipeline as described in the notebook on selecting filter pipelines.
For all of below examples, we need to load at least one data set which we will use to interactively preview our filter settings. Note that for a good interactive experience with no downtimes, you should restrict your datasets to a reasonable size (see the Working with datasets notebook for how to do it). Loading multiple datasets might be beneficial to avoid overfitting the filtering pipeline to one given dataset.
[2]:
dataset = afwizard.DataSet(
filename="nkd_pcl_epsg6670.laz", spatial_reference="EPSG:6670"
)
Creating from scratch
The main pipeline configuration is done by calling the pipeline_tuning function with your dataset as the parameter. This will open the interactive user interface which allows you to tune the filter pipeline itself in the left column and the visualization and rasterization options in the right column. Whenever you hit the Preview button, a new tab will be added to the center column. Switching between these tabs allows you to switch between different version of your filter. The return object
pipeline is updated on the fly until you hit the Finalize button to freeze the currently displayed filter.
[3]:
pipeline = afwizard.pipeline_tuning(dataset)
---------------------------------------------------------------------------
AFwizardError Traceback (most recent call last)
Cell In[3], line 1
----> 1 pipeline = afwizard.pipeline_tuning(dataset)
File ~/checkouts/readthedocs.org/user_builds/afwizard/conda/stable/lib/python3.11/site-packages/afwizard/apps.py:304, in pipeline_tuning(datasets, pipeline)
302 # Instantiate a new pipeline object if we are not modifying an existing one.
303 if pipeline is None:
--> 304 pipeline = Pipeline()
306 # If a single dataset was given, transform it into a list
307 if isinstance(datasets, DataSet):
File ~/checkouts/readthedocs.org/user_builds/afwizard/conda/stable/lib/python3.11/site-packages/afwizard/filter.py:371, in PipelineMixin.__init__(self, _variability, **kwargs)
368 filters.append(f)
369 kwargs["filters"] = filters
--> 371 self.config = kwargs
372 self.variability = _variability
File ~/checkouts/readthedocs.org/user_builds/afwizard/conda/stable/lib/python3.11/site-packages/afwizard/filter.py:79, in Filter.config(self, _config)
76 # Validate the given configuration
77 _config = pyrsistent.freeze(_config)
78 jsonschema.validate(
---> 79 instance=pyrsistent.thaw(_config), schema=pyrsistent.thaw(self.schema())
80 )
82 # Store the validated config
83 self._config = _config
File ~/checkouts/readthedocs.org/user_builds/afwizard/conda/stable/lib/python3.11/site-packages/afwizard/filter.py:401, in PipelineMixin.schema(cls)
399 @classmethod
400 def schema(cls):
--> 401 return cls._schema_impl("schema")
File ~/checkouts/readthedocs.org/user_builds/afwizard/conda/stable/lib/python3.11/site-packages/afwizard/filter.py:383, in PipelineMixin._schema_impl(cls, method)
381 for ident, class_ in Filter._filter_impls.items():
382 if Filter._filter_is_backend[ident]:
--> 383 if class_.enabled():
384 bschema = getattr(class_, method)()
385 backend_schemas.append(pyrsistent.thaw(bschema))
File ~/checkouts/readthedocs.org/user_builds/afwizard/conda/stable/lib/python3.11/site-packages/afwizard/lastools.py:139, in LASToolsFilter.enabled(cls)
137 @classmethod
138 def enabled(cls):
--> 139 return lastools_is_present()
File ~/checkouts/readthedocs.org/user_builds/afwizard/conda/stable/lib/python3.11/site-packages/afwizard/lastools.py:69, in lastools_is_present()
66 return False
68 # We return True iff a prefix was set
---> 69 return get_lastools_directory() is not None
File ~/checkouts/readthedocs.org/user_builds/afwizard/conda/stable/lib/python3.11/site-packages/afwizard/lastools.py:55, in get_lastools_directory()
53 dir = os.environ.get("LASTOOLS_DIR", None)
54 if dir is not None:
---> 55 set_lastools_directory(dir)
57 return _lastools_directory
File ~/checkouts/readthedocs.org/user_builds/afwizard/conda/stable/lib/python3.11/site-packages/afwizard/lastools.py:44, in set_lastools_directory(dir)
42 except AFwizardError as e:
43 _lastools_directory = None
---> 44 raise e
File ~/checkouts/readthedocs.org/user_builds/afwizard/conda/stable/lib/python3.11/site-packages/afwizard/lastools.py:41, in set_lastools_directory(dir)
38 if dir is not None:
39 try:
40 # If this throws, we show a meaningful error where we looked for LASTools
---> 41 lasground_executable(base=dir)
42 except AFwizardError as e:
43 _lastools_directory = None
File ~/checkouts/readthedocs.org/user_builds/afwizard/conda/stable/lib/python3.11/site-packages/afwizard/lastools.py:84, in lasground_executable(base)
82 fullpath = os.path.join(base, "bin", execname)
83 if not os.path.exists(fullpath):
---> 84 raise AFwizardError(f"Executable {fullpath} was not found!")
86 return fullpath
AFwizardError: Executable /home/docs/checkouts/readthedocs.org/user_builds/afwizard/checkouts/latest/LAStools/bin/lasground_new64.exe was not found!
If you want to inspect multiple data sets in parallel while tuning a pipeline, you can do so by passing a list of datasets to the pipeline_tuning function. Note that AFwizard does currently not parallelize the execution of filter pipeline execution which may have a negative impact on wait times while tuning with multiple parameters. A new tab in the center column will be created for each dataset when clicking Preview:
[4]:
pipeline2 = afwizard.pipeline_tuning(datasets=[dataset, dataset])
---------------------------------------------------------------------------
AFwizardError Traceback (most recent call last)
Cell In[4], line 1
----> 1 pipeline2 = afwizard.pipeline_tuning(datasets=[dataset, dataset])
File ~/checkouts/readthedocs.org/user_builds/afwizard/conda/stable/lib/python3.11/site-packages/afwizard/apps.py:304, in pipeline_tuning(datasets, pipeline)
302 # Instantiate a new pipeline object if we are not modifying an existing one.
303 if pipeline is None:
--> 304 pipeline = Pipeline()
306 # If a single dataset was given, transform it into a list
307 if isinstance(datasets, DataSet):
File ~/checkouts/readthedocs.org/user_builds/afwizard/conda/stable/lib/python3.11/site-packages/afwizard/filter.py:371, in PipelineMixin.__init__(self, _variability, **kwargs)
368 filters.append(f)
369 kwargs["filters"] = filters
--> 371 self.config = kwargs
372 self.variability = _variability
File ~/checkouts/readthedocs.org/user_builds/afwizard/conda/stable/lib/python3.11/site-packages/afwizard/filter.py:79, in Filter.config(self, _config)
76 # Validate the given configuration
77 _config = pyrsistent.freeze(_config)
78 jsonschema.validate(
---> 79 instance=pyrsistent.thaw(_config), schema=pyrsistent.thaw(self.schema())
80 )
82 # Store the validated config
83 self._config = _config
File ~/checkouts/readthedocs.org/user_builds/afwizard/conda/stable/lib/python3.11/site-packages/afwizard/filter.py:401, in PipelineMixin.schema(cls)
399 @classmethod
400 def schema(cls):
--> 401 return cls._schema_impl("schema")
File ~/checkouts/readthedocs.org/user_builds/afwizard/conda/stable/lib/python3.11/site-packages/afwizard/filter.py:383, in PipelineMixin._schema_impl(cls, method)
381 for ident, class_ in Filter._filter_impls.items():
382 if Filter._filter_is_backend[ident]:
--> 383 if class_.enabled():
384 bschema = getattr(class_, method)()
385 backend_schemas.append(pyrsistent.thaw(bschema))
File ~/checkouts/readthedocs.org/user_builds/afwizard/conda/stable/lib/python3.11/site-packages/afwizard/lastools.py:139, in LASToolsFilter.enabled(cls)
137 @classmethod
138 def enabled(cls):
--> 139 return lastools_is_present()
File ~/checkouts/readthedocs.org/user_builds/afwizard/conda/stable/lib/python3.11/site-packages/afwizard/lastools.py:69, in lastools_is_present()
66 return False
68 # We return True iff a prefix was set
---> 69 return get_lastools_directory() is not None
File ~/checkouts/readthedocs.org/user_builds/afwizard/conda/stable/lib/python3.11/site-packages/afwizard/lastools.py:55, in get_lastools_directory()
53 dir = os.environ.get("LASTOOLS_DIR", None)
54 if dir is not None:
---> 55 set_lastools_directory(dir)
57 return _lastools_directory
File ~/checkouts/readthedocs.org/user_builds/afwizard/conda/stable/lib/python3.11/site-packages/afwizard/lastools.py:44, in set_lastools_directory(dir)
42 except AFwizardError as e:
43 _lastools_directory = None
---> 44 raise e
File ~/checkouts/readthedocs.org/user_builds/afwizard/conda/stable/lib/python3.11/site-packages/afwizard/lastools.py:41, in set_lastools_directory(dir)
38 if dir is not None:
39 try:
40 # If this throws, we show a meaningful error where we looked for LASTools
---> 41 lasground_executable(base=dir)
42 except AFwizardError as e:
43 _lastools_directory = None
File ~/checkouts/readthedocs.org/user_builds/afwizard/conda/stable/lib/python3.11/site-packages/afwizard/lastools.py:84, in lasground_executable(base)
82 fullpath = os.path.join(base, "bin", execname)
83 if not os.path.exists(fullpath):
---> 84 raise AFwizardError(f"Executable {fullpath} was not found!")
86 return fullpath
AFwizardError: Executable /home/docs/checkouts/readthedocs.org/user_builds/afwizard/checkouts/latest/LAStools/bin/lasground_new64.exe was not found!
Storing and reloading filter pipelines
Pipeline objects can be stored on disk with the save_filter function from AFwizard. The filename passed here, can either be an absolute path or a relative one. Relative paths are interpreted w.r.t. the current working directory unless a current filter library has been declared with set_current_filter_library:
[5]:
afwizard.save_filter(pipeline, "myfilter.json")
---------------------------------------------------------------------------
NameError Traceback (most recent call last)
Cell In[5], line 1
----> 1 afwizard.save_filter(pipeline, "myfilter.json")
NameError: name 'pipeline' is not defined
The appropriate counterpart is load_filter, which restores the pipeline object from a file. Relative paths are interpreted w.r.t. to the filter libraries known to AFwizard:
[6]:
old_pipeline = afwizard.load_filter("myfilter.json")
---------------------------------------------------------------------------
TypeError Traceback (most recent call last)
Cell In[6], line 1
----> 1 old_pipeline = afwizard.load_filter("myfilter.json")
File ~/checkouts/readthedocs.org/user_builds/afwizard/conda/stable/lib/python3.11/site-packages/afwizard/filter.py:543, in load_filter(filename)
540 # Find the file across all libraries
541 from afwizard.library import locate_filter
--> 543 filename = locate_filter(filename)
545 with open(filename, "r") as f:
546 return deserialize_filter(json.load(f))
File ~/checkouts/readthedocs.org/user_builds/afwizard/conda/stable/lib/python3.11/site-packages/afwizard/library.py:226, in locate_filter(filename)
223 return nakadake_data.fetch(filename)
225 # Maybe this is a filter shipped as part of our testing data
--> 226 if os.path.exists(download_test_file(filename)):
227 return download_test_file(filename)
229 # If we have not found it by now, we throw an error
File <frozen genericpath>:19, in exists(path)
TypeError: stat: path should be string, bytes, os.PathLike or integer, not NoneType
A filter pipeline loaded from a file can be edited using the pipeline_tuning command by passing it to the function. As always, the pipeline object returned by pipeline_tuning will be a new object - no implicit changes of the loaded pipeline object will occur:
[7]:
edited_pipeline = afwizard.pipeline_tuning(dataset, pipeline=old_pipeline)
---------------------------------------------------------------------------
NameError Traceback (most recent call last)
Cell In[7], line 1
----> 1 edited_pipeline = afwizard.pipeline_tuning(dataset, pipeline=old_pipeline)
NameError: name 'old_pipeline' is not defined
Batch processing in filter creation
The pipeline_tuning user interface has some additional powerful features that allow you to very quickly explore parameter ranges for filter. You can use this feature by clicking the symbol next to a parameter. This will open a flyout where you can specify a range of parameters to generate previews for. Ranges can either be a discrete comma separated list e.g. 1, 2, 3, a range of parameters like 4:6 or a mixture there of. Ranges are only available for numeric inputs and can be
provided an optional increment after a second colon like e.g. 1:5:2. In the absence of an explicit increment, integer ranges use an increment of 1 and float ranges sample the range with a total of 5 samples points. When clicking Preview, batch processing information is resolved and the batch information is discarded.
Filter pipelines with end user configuration
The goal in creation of filter pipelines in AFwizard is to provide pipelines that are on the one hand specialized to a given terrain type and on the other hand generalize well to other datasets of similar terrain. In order to achieve this it is sometimes necessary to define some configuration values that are meant to be finetuned by the end user. We can do by clicking the symbol next to a parameter. Like in batch processing, a flyout opens where we can enter values, a display name for the
parameter and a description. Values can either be a comma-separated list of values or a single range of parameters with a :. These parameters are displayed to the end user when selecting a fitting filter pipeline as described in Selecting a filter pipeline for a dataset. This end user configuration interface can also be manually invoked by using the filter pipeline’s execute_interactive method:
[8]:
tuned = pipeline.execute_interactive(dataset)
---------------------------------------------------------------------------
NameError Traceback (most recent call last)
Cell In[8], line 1
----> 1 tuned = pipeline.execute_interactive(dataset)
NameError: name 'pipeline' is not defined
Applying filter pipelines to data
Pipeline objects can also be used to manipulate data sets by applying the ground point filtering algorithms in a non-interactive fashion. This is one of the core tasks of the afwizard library, but this will rarely be done in this manual fashion, as we will provide additional interfaces for (locally adaptive) application of filter pipelines:
[9]:
filtered = pipeline.execute(dataset)
---------------------------------------------------------------------------
NameError Traceback (most recent call last)
Cell In[9], line 1
----> 1 filtered = pipeline.execute(dataset)
NameError: name 'pipeline' is not defined
The returned object is a dataset object in itself that can again be treated like described in Working with datasets:
[10]:
filtered.show_interactive()
---------------------------------------------------------------------------
NameError Traceback (most recent call last)
Cell In[10], line 1
----> 1 filtered.show_interactive()
NameError: name 'filtered' is not defined