You are currently viewing Streamlining agriculture operations with serverless anomaly detection utilizing AWS IoT

Streamlining agriculture operations with serverless anomaly detection utilizing AWS IoT


Honeybees reside in swarms of tens of 1000’s, gathering nectar. On this course of, they carry pollen from one flowering plant to a different, pollinating them.

” Near 75 % of the world’s crops producing fruits and seeds for human use rely, not less than partly, on pollinators[1]. ”

In addition to being one in all nature’s key pollinators, bees remodel nectar into honey. With the assistance of beekeepers, like David Gerber from Switzerland, this scrumptious honey is made accessible for international consumption.

David Gerber’s IoT enabled beehives (Neuchatel, Switzerland)

Determine 1: David Gerber’s IoT enabled beehives (Neuchatel, Switzerland)

Bees reside in hives. These hives are sometimes situated in distant areas, like forests or excessive mountain pastures. These distant areas make monitoring the well being of bees difficult. Nevertheless, by creating linked options utilizing cloud-based providers, reminiscent of AWS IoT Core and AWS Lambda, beekeepers can implement close to real-time monitoring instruments to trace well being parameters for a bee hive. AWS IoT Core is a totally managed cloud service, that allows you to join Web of Issues (IoT) gadgets and route their messages to AWS with out managing infrastructure. AWS Lambda is a serverless compute service permitting you to deploy code with out provisioning or deploying servers. On this weblog put up, we stroll by means of an IoT structure and supply a hands-on instance of find out how to create and take a look at your personal serverless anomaly detector to enhance your operations.


For this stroll by means of, it is best to have the next conditions:

The hands-on instance is written in Java and the CDK infrastructure code is written in Typescript. It’s not required to have deep information in both to deploy and run the instance. This resolution can run totally inside the AWS Free Tier for one and even a number of executions. Clear-up directions are offered on the finish of this put up.

Gaining insights into hive well being

We achieve insights by measuring and sending IoT occasions. Selecting what to measure a couple of hive is vital. The appropriate metric permits us to achieve insights into the lives of the bees. In Determine 2, we will see the variation of a hive’s weight as the times go by. At first look, the information seems fairly chaotic. Nevertheless, a more in-depth look reveals a wealth of knowledge.

Figure 2 : Weight of hive over two weeks

Determine 2 : Weight of hive over two weeks

From Determine 2, we will chart a hive’s main occasions over 24 hours.

  1. Bees make honey by lowering the nectar’s water content material. Bees fan their wings to create airflow inside the hive, inflicting the water within the nectar to evaporate. This leads to a gradual weight discount of the hives in a single day.
  2. At dawn, the bees are prepared for his or her day’s work, inflicting a sudden drop within the hive’s weight.
  3. Over the day, bees return to the hive carrying nectar with them, inflicting a gradual enhance within the weight of the hive.
  4. At sundown, all of the bees return to the hive with their remaining sector leading to a sudden enhance within the hive’s weight.
  5. Lastly, by evaluating the hive’s weight, on the similar time of day 24 hours aside, we will inform how productive the hive has been.
Figure 3 A hive's major events over 24 hours

Determine 3 A hive’s main occasions over 24 hours

Detecting anomalies

Returning to the unique dataset in Determine 3, we will see, the truth is, the primary week has been very productive :

  1. The bees profit from glorious circumstances with a every day enhance within the hive’s weight.
  2. The beekeeper extracts roughly 10kg of honey on the finish of the week.
Figure 4 : Daily increase in hives weight

Determine 4 : Each day enhance in hives weight

Nevertheless, not each week is pretty much as good, and initially of the second week in Determine 4, we will see issues get off to a tougher begin.

  1. The bees don’t depart the hive; this might be as a result of a scarcity of nectar within the space, an indication to contemplate transferring the hive.
  2. Or it might be simply short-term unhealthy climate, which passes and permits the bees to proceed gathering nectar in a while within the week.
Figure 5  : First week of June

Determine 5  : First week of June

After taking a sequence of measurements, an anomaly deviates from what we’ve beforehand seen; it’s surprising. Dangerous climate could be detected as an anomalous occasion, however little could be executed. Sadly, each bees and people need to reside with it. Nevertheless, a number of different anomalous occasions could be useful to detect in distant hives.

  1. A sudden enhance within the amount of nectar accessible for bees to gather leads to a major rise in honey manufacturing referred to as honeyflow.
    1. Throughout a honeyflow, the burden of a hive can enhance every day by a kilogram and lets the beekeeper realize it’s time so as to add further area to the hive.
    2. Conversely, a stagnation in weight will increase permits the beekeeper to substantiate the top of the honeyflow. The honey can be accessible to reap a couple of days later after its moisture content material has been lowered.
  2.  A sudden enhance within the every day sector collected over a 24-hour interval lets the beekeeper realize it’s time to gather the honey and unlock area to permit the bees to proceed working.
  3. When a hive grows, it’s going to finally cut up in two by swarming, with half the hive deciding to go away (a sudden lower in weight however not at dawn) with the previous queen. Usually, this swarm will settle in a brief location and could be recaptured by the beekeeper if detected in time.
  4. A big discount in weight of tens of kilos implies somebody apart from the beekeeper is gathering the honey, resulting in potential operational losses for beekepers.

Answer overview

Figure 6 : The overall AWS architecture of the solution

Determine 6 : The general AWS structure of the answer

Determine 6 reveals the general AWS structure of the answer. The answer makes use of IoT sensors deployed below every beehive to ship the hive’s weight recurrently in an IoT occasion. These IoT sensors talk utilizing the LoRaWAN  protocol. LoRaWAN is ideally suited to the supply of IoT occasions in hard-to-reach areas. It trades severely proscribing message payload measurement for the power to ship this payload over kilometers utilizing minimal energy consumption. The beehive’s IoT sensors sends the occasion to a Issues Community (TTN) Gateway. TTN democratizes entry to an IoT community, permitting individuals to arrange their personal gateways. This gateway is the communication hyperlink between the IoT sensor and AWS IoT Core for LoRaWAN. AWS IoT Core for LoRaWAN supplies entry to a totally managed LoRaWAN Community Server (LNS), eliminating the necessity to develop, keep, or function a separate server. You will discover additional particulars on integrating TTN and AWS IoT Core right here.

Utilizing AWS IoT Core Guidelines Engine, you’ll be able to routinely route messages to Amazon Easy Queue Service (Amazon SQS). This decouples AWS IoT Core from AWS Lambda, permitting the IoT occasion to be processed asynchronously. AWS Lambda permits the anomaly detection code to be deployed in a serverless vogue, eliminating, but once more, the necessity to handle your infrastructure. AWS Lambda will scale horizontally to fulfill any enhance in IoT visitors. The primary of two Lambda features persists the occasion and permits all earlier occasions to be sorted on retrieval. Retrieval of occasions in chronological order is crucial in figuring out whether or not an occasion is anomalous.

The anomaly detection code operating in AWS Lambda lies on the coronary heart of the answer. It depends on an implementation of the Random Minimize Forest (RCF) [2] algorithm written by AWS. RCF is a machine studying algorithm able to detecting anomalies in an unsupervised method. The algorithm constructs collections of random binary bushes. An anomaly rating displays how far some extent is from the others within the tree. Outlying knowledge factors are much less prone to be per different knowledge factors within the tree, resulting in greater anomaly scores. RCF is designed to course of streamed multi-dimensional knowledge effectively, making it excellent for our situation of streamed IoT messages containing the beehive’s weight. Lastly, the beekeeper could be notified of anomalous occasions utilizing Amazon Easy Notification Service.

Palms-on setup structure

Figure 7 : Simulation architecture

Determine 7 : Simulation structure

To check the anomaly detection resolution extra simply from our laptops, we’ve created a 3rd Lambda perform, which can simulate the creation of IoT occasions throughout Could (see Determine 7).

Figure 8 : Simulation data

Determine 8 : Simulation knowledge

Determine 8 visualizes the artificial knowledge used for the simulation.  The info reveals a gradual enhance within the hive’s weight over thirty days ranging from the first of Could. The hive’s weight peaks every night whereas step by step lowering in weight in a single day, with a sudden dip because the hive departs at dawn. The hive’s weight slowly recovers through the day with the return of nectar-laden bees. The info set comprises 720 knowledge factors (30 days occasions 24 hours). Just one knowledge level is uncommon: the eighth of Could, when the hive’s weight is unexpectedly lowered by 1.5+ Kg. This instance reveals the facility of the RCF algorithm; a easy threshold worth is not going to suffice as a result of hives growing weight. Certainly the eighth of Could anomaly is a sound knowledge level on the morning of the 4th of Could.

Simulation execution and outcomes

The aim of the simulation is to appropriately determine the one anomalous IoT occasion (on the eighth of Could at 04:00) among the many 719 different occasions. Please check with the beehive-anomaly-detection-simulation git repository for extra particulars on surroundings setup and directions on how one can run the simulation out of your laptop computer.

  1. Earlier than we deploy any infrastructure, we first need to compile and bundle the Java Lambda by operating the next instructions:
git clone
cd iot-beehive-anomaly-detection-simulation-blog-source-code
mvn clear set up
  1. The infrastructure for this simulation is described utilizing AWS Cloud Growth Equipment (CDK). CDK lets you outline every infrastructure element as code, in our case, utilizing typescript.
const iotEventsSQSQueue = new sqs.Queue(this, 'IoTEventsSQSQueue', {
    visibilityTimeout: cdk.Length.seconds(120),
    queueName: 'iot-events'

new iot.TopicRule(this, 'IoTEventsSQSQueueRule', {
    topicRuleName: 'ioTEventsSQSQueue',
    description: 'invokes the lambda perform',
    sql: iot.IotSql.fromStringAsVer20160323("SELECT * FROM 'iot/beehive'"),
    actions: [new actions.SqsQueueAction(iotEventsSQSQueue)],

For instance, within the code snippet above, we describe the creation of an SQS queue named iot-events and an AWS IoT Core rule that forwards IoT occasions from the iot/beehive MQTT subject to the SQS queue. Equally, all of the remaining infrastructure parts (the three Lambdas and one DynamoDB desk) are outlined in infrastructure/lib/infrastructure-stack.ts

We deploy the infrastructure utilizing the next CDK instructions. If that is the primary time you deploy infrastructure with CDK, you will want to bootstrap. CDK bootstrapping units up permissions insurance policies, an AWS CloudFormation stack, and an S3 bucket to retailer deployment property. It’s required solely as soon as per account and area.

Run the next instructions to deploy our infrastructure:

cd infrastructure
npm set up
cdk bootstrap
cdk deploy
  1. Now, we will start the simulation correct by invoking the IoTBeehiveEventsSimulator. On the core of this Lambda, we create an AWSIotDataAsyncClient, a shopper for accessing the AWS IoT Information airplane asynchronously. For each factor in the iot-beehive-events-simulator-lambda/src/most important/sources/hive-sample-events.json array an IoT occasion is shipped to the MQTT subject iot/beehive. The standard of service (QoS) is ready to 1, guaranteeing the occasion is shipped not less than as soon as. As we can’t assure precisely as soon as occasion supply in distributed programs, the selection is between not receiving an occasion or receiving an occasion a number of occasions. Nevertheless, we will guarantee precisely as soon as processing by making every Lambda idempotent. They return the identical outcome whether or not they’re referred to as as soon as or many occasions.
AWSIotData iotClient = AWSIotDataAsyncClientBuilder.defaultClient();

for (HiveEvent hiveEvent : hiveEvents) {
    PublishRequest publishRequest = new PublishRequest()

Run the next command to start the simulation:

aws lambda invoke --function-name IoTBeehiveEventsSimulatorLambda --cli-binary-format raw-in-base64-out --payload '{"hiveID":"1"}' response.json

We are able to verify that every one IoT occasions have been persevered efficiently by operating a full scan of the DynamoDB desk with the next command and guaranteeing the result’s 720.

aws dynamodb scan --table-name HIVE_EVENTS --select "COUNT"

Notice: Be happy to name IoTBeehiveEventsSimulator a number of occasions, confirming every distinctive occasion is processed precisely as soon as.

  1. Lastly, it’s time to find out if any IoT occasions are anomalous by operating IoTAnomalyDetectionLambda. The anomaly detection Lambda reads the IoT occasions from a DynamoDB desk. DynamoDB is crucial in guaranteeing no occasions are misplaced and permits the processing of IoT occasions so as (in response to their timestamp). Whether or not the hive weight at any specific time limit is as anticipated can solely be decided by an ordered processing of earlier occasions.

Run the next instructions to start the anomaly detection. The outcomes are saved within the response.json file:

aws lambda invoke --function-name IoTAnomalyDetectionLambda --cli-binary-format raw-in-base64-out --payload '{"hiveID": "1"}' response.json
much less response.json | jq

Pattern Response:

    "datetime": "2023-05-08 04:00:00.0 +0200",
    "weight": 64650,
    "anomalyGrade": 1.0,
    "anomalyScore": 1.2257463093204803,
    "expectedValue": 66195,
    "isEventAnomalous": true

An anomaly rating represents how doubtless the occasion is to be an outlier, with a threshold worth of 1.0 usually used to indicate an anomaly. The rating of a mannequin and its (inverse remodel to) inference are thought-about individually. Therefore, we’ve an anomaly grade. In our case the transformation is a normalization of the occasion stream, the place the linear enhance in weight of the hive is factored out. An anomaly grade ranges from 0 to 1, the place a price higher than 0 is doubtless anomalous.

Figure 9: Successful detection of anomaly

Determine 9: Profitable detection of anomaly

In determine 9 we will see the CloudWatch metrics reported by the anomaly detection algorithm present certainly, solely a single anomaly has been detected. Moreover, the response confirms the anomalous occasion is from 04:00 on the eighth of Could.

Calculating an occasion’s anomaly detection by reprocessing the earlier occasions saved in DynamoDB provides a number of seconds of latency to the rating calculation. Nevertheless, this enables the answer to stay totally serverless, making it a suitable trade-off. Streaming the occasions utilizing Amazon Managed Service for Apache Flink might be an alternate resolution for latency-sensitive options.

Cleansing up

Infrastructure created with CDK could be very simply torn down. Merely run the next instructions from a terminal.

cd infrastructure
cdk destroy


The weblog put up confirmed how IoT can remedy thrilling and vital challenges within the pure world. The structure we offered is totally serverless, conserving prices and infrastructure upkeep efforts low. Lastly, we walked by means of a hands-on instance the place you’ll be able to dive into the code and run the simulations your self. If you wish to work by yourself IoT tasks, try TTN and AWS IoT.




David Gerber.jpg

David Gerber

David works together with his buyer’s groups on their full software program growth lifecycle, from preliminary ideas proper by means of to manufacturing. He’s keen about software program growth, IoT and … beekeeping.

Kevin Nash.jpg

Kevin Nash

Kevin is a Senior Options Architect at Amazon Net Companies (AWS), based mostly in Switzerland. With a background in distributed programs and a few years expertise constructing for the client. He’s keen about expertise, understanding how programs work and serving to clients bringing their options into the Cloud.

Leave a Reply