What is big data and how to deal with it

The term “big data” is often defined with reference to the characteristics of the volume, variety and velocity of data (the “three V’s”).

The characteristic of “volume” refers to the quantity or magnitude of data. “Variety” refers to the range of different types of data. “Velocity” refers to the speed at which data is generated, processed or analysed.

Other features of big data that have been identified are veracity, variability and complexity. The characteristic of “veracity” refers to the unreliability or imprecision of certain data. “Variability” refers to variability in the rate or velocity of data flow. “Complexity” refers to the multiple sources from which data may be generated.

Big data is created, transferred, stored, hosted, used and processed daily in virtually all industry and community sectors, including by government, the private sector and other community organisations.

Certain types of data may require specific consents, approvals, licences, or agreements in order to permit an organisation to perform certain acts in relation to such data.

Big data may consist of personal information, data protected by copyright, data protected by moral rights, confidential information, and government official information.

Where an organisation enters into a commercial agreement with another organisation that involves the capture, receipt, transfer, hosting, processing or management of big data, there are a number of issues to consider, including which party is responsible for obtaining necessary consents, what contractual obligations will apply in relation to the big data, and technical, functional and performance specifications in relation to dealing with big data.