How to manage your unstructured drone-collected data

Drones are excellent for collecting information from above. But finding a way to store that data is more of a challenge.

Drones are excellent for collecting information from above. But finding a way to store that data is more of a challenge.

Flying drone over the rapeseed field at spring, dramatic sunset clouds
Image: valio84sl, Getty Images/iStockphoto

More about Big Data

When a Danish company wanted to perform underwater surveys of several shallow ponds spread across the country, it soon found that the thickness of the seaweed in the ponds rendered it impossible to conduct the underwater surveys with standard submersible echo sounding equipment that bounces sonar waves off seabed sand and underwater objects.

SEE: Tableau business analytics platform: A cheat sheet (free PDF download) (TechRepublic)

The solution was a drone, which emitted ground penetrating radar (GPR) that was able to do the same job without the problem of having to be submerged through seaweed.

Use cases like this for drones and big data collection are surfacing in virtually every industry sector—from traffic monitoring and accident documentation to agricultural inspections and forest wildfire fighting.

In each case, data is most often collected in the form of photos and videos. The data can be stored on the drone and then relayed back to a ground-based data repository where it can be studied and analyzed.

The process seems straightforward enough, but for IT and end-business departments running drones, new protocols, business processes, and data practices need to be in place.

SEE: Inside UPS: The logistics company’s never-ending digital transformation (free PDF) (TechRepublic)

If you are charged with collecting big data from drones, here are six questions that you should ask yourself:

1. Where’s the data?

Because of bandwidth and location constraints, few sites want to be downloading their drone data to a distant central repository in real time, or even in batch immediately following flight times. The storage architecture most often deployed is remote storage in the cloud, or at a secure field office proximate to drone activities. Data can be transferred to a central data repository later. What IT must do is to define this drone data architecture and how it will work in advance of any drone activity.

2. Is the data secure?

Drones collect data in the field, and the data is usually housed in remote offices. In these remote offices, it is more than likely that a non-IT person will be in charge. Standard IT security and custodial practices need to be put in place and trained to these non-IT employees so data and data storage remain physically and logically secure.

3. How much data do you want?

“If you’re flying three to five times a week and averaging 500 images per flight, you’re amassing tens of gigabytes each week and you need to make a plan about what’s going to happen with all this info, and specific choices in this area depend on what you need to do,” said Benjamin Van der Jagt, co-founder of UMAP Technologies, a 3D mapping startup.

The data is the point. Early on, IT and business users should decide which data they want to keep and which they will discard. If you fail to make these determinations, the data will overrun your storage capacity and you’ll later find it even more difficult to sort through all of the data to find the information you really want.

4. What is the impact on storage?

Storage can be relatively cheap, but having to maintain rows of servers and storage drives with data that is never or seldom used is not what you want. That’s why early on in the planning of a drone project, you want to determine which data you want to keep from drone missions; and how much storage you’re going to have to deploy or fund to store all of that data.

5. What happens if data is lost?

Drones have limited flight times because their batteries have limits. These flight times can be further reduced if unexpected headwinds are encountered. Then, too, a drone could just fail—or it could be intercepted. Companies choosing to run drones should have a plan of action for recovering lost drones (and potentially sensitive data) or for locking drone data down if the machine can’t be recovered.

6. Can drone data be integrated with data from other systems for advanced analytics?

Drone-collected data is almost always in some unstructured video or photographic format. This doesn’t make it easy to integrate with data from other corporate systems that are character-based and in the form of fixed-length records.

A great platform that effectively integrates with drone-collected data is a geographic information system (GIS). GIS systems are primarily used for mapping. They can support and integrate non-textual information with GPS and other types of data, and are an ideal integration platform for drone-collected data.

SEE: Census tech: How to count every person (TechRepublic)

Unfortunately, drone and GIS data data isn’t easily integrated with data from other corporate systems. In some cases, IT may choose to invest in a GIS system if it doesn’t already have one. In other cases, IT may choose to pursue more wide ranging—and difficult—data integration if business cases and analytics warrant it.