Why Does Large-scale Data Impede Spatial Operations?

Storage -

“  Data is a rapidly accelerating machine, with billions of data points being created every single hour. As data continues to flow from all areas of the world, it’s no surprise that geography has also become a potent source. “

- Geospatial World 

• Massive amounts of spatial data are generated from sources like social media platforms, mobile apps, and location-based services, including user check-ins, geotagged posts, and location history.

• The rise of connected vehicles produces real-time GPS and telemetry data, while IoT sensors and satellite images further contribute to this vast data load.

• Managing and storing large volumes of spatial data, which can range from gigabytes to petabytes, requires robust infrastructure, storage solutions, and computational resources. Not all companies may be willing or able to invest significantly in infrastructure upgrades or cloud services to handle spatial data effectively.

Computation - 

• The compute power required for handling spatial data depends on various factors such as the volume of data, complexity of spatial operations, desired performance, and the specific tasks involved. 

• Basic operations like filtering on spatial data may be less intensive than complex ones like spatial join and enrichment.

• Due to the high computing resource requirements for efficient implementation of spatial operations, companies often face increased compute costs, particularly in cloud environments where resources are charged based on usage.

Why is the Available Data Quality Low?

“ Business decisions are only as good as the data you use to make them “

Acquiring meaningful and high-quality spatial data can be a daunting task. Furthermore, even after gaining access to the data, conducting thorough quality checks to verify its reliability presents additional difficulties.

Resolution:

• Limitations around major platform ecosystems have constrained data accessibility. Over the past few years, both Apple and Google have implemented measures that restrict the sharing of advertising identifiers (IDFA) and approximate location data to a 10-square-mile radius. This has significantly reduced the amount of usable location data accessible.\

  
  

• Human mobility patterns follow a power law distribution, characterized by a minority of users with high activity levels and a majority with sporadic activity.  Analyzing consumers at a granular level is thus difficult at scale due to statistical challenges.
  
  

Privacy Concerns:

• Recent regulations like GDPR and CPRA impose strict requirements on the collection, processing, and use of personal data, including location information. 
  
  

• Strict privacy rules make it tough to gather precise location info without clear consent from individuals.
  
  

Adversarial Actors:

• Dishonest actors in Ad Tech are faking location data using techniques like IP geocoding or randomly assigning coordinates.
  
  

• These fake data tactics lower the overall quality of location information available for legitimate use.

What hinders current platforms from fully solving the above questions?

Prohibitive Costs -  Running queries and analyses on spatial data at scale can be prohibitively expensive, especially on existing platforms which drain the compute resources for large datasets so companies just decide not to execute all the necessary queries, instead resort to sampling and other methods, leading to suboptimal outcomes.

Lack of spatial capability - The current compute platforms, such as cloud warehouse solutions, are primarily designed for general-purpose computing tasks, which poses challenges for non-GIS experts when running spatial data workflows.They lack support for managing multimodal data and aiding data teams, consequently constraining their analysis to familiar datasets and data types.

The Future Potential 

In essence, although spatial data holds considerable value for data analysis across various scenarios, its widespread utilization remains limited due to the inherent complexity of the data and the absence of computing platforms capable of handling such intricacies.

This offers an opportunity for emerging platforms to handle the complexities and extract value from the data, thus enhancing its usefulness in a broader range of analyses for various user types across diverse industries.

In the next post of this blog series, we will delve into the opportunities we have identified and present our framework aimed at addressing these challenges in detail.