Criteria

Phase 1: Creator

A.  Phase 1 Contestants will stand in the shoes of experts trying to understand and defeat robocaller tactics. Contestants will be required to: 1) build a honeypot that collects data on each call; and 2) categorize the calls based on the likelihood that the call is a robocall (a call delivering a prerecorded message).  In designing the honeypot, Contestants may not include any feature that requires ongoing manual processing. Each Contestant will receive free access to a Twilio account, which requires an agreement to comply with Twilio’s terms of service. Twilio will provide a platform that Contestants may use to build their honeypot and at least one phone number associated with the Twilio account.  If Contestants build a honeypot on Twilio’s platform, they will also receive $30 of credit. Without spending any credit, each Contestant may receive and send calls to the phone number associated with his or her Twilio account from one external phone number. To send and receive calls using additional lines, Contestants may spend up to their $30 credit limit. The credit may also be applied toward other Twilio features.  Contestants, however, are not required to use the Twilio platform and may use any tools they choose so long as Contestants also provide one or more honeypot phone lines, access to the associated call detail records or call logs, and any other information so that the Judges can test their Submissions without additional cost.

Contestants will be disqualified, however, if they: 1) place calls using an autodialing program or the functional equivalent; 2) adversely affect the Twilio platform; 3) adversely affect any other platform, product, system, or technology; 4) violate the terms of service of Twilio or any other third party provider; 5) do anything prohibited by law; or 6) place calls to any number outside their own honeypot. Judges will test the Submissions by sending calls to the honeypots and reviewing the information associated with incoming calls. Each Contestant will submit all source code in addition to a written description of the solution, consisting of fewer than 500 words, summarizing the Contestant’s techniques and outcomes.

Judging Criteria:

(i) Building Knowledge (70% of total score)

• Did you succeed in identifying inaccuracies in the data captured?  You will receive ten points for each distinct successful method that your honeypot used to identify calls in which the data captured in any field was inaccurate. Inaccurate data could include, but is not limited to, false caller ID number or date of the call. Contestants will need to prove to the Judges’ satisfaction that the data captured was inaccurate, and the necessary level of proof is within the Judges’ sole discretion. Furthermore, the Judges have sole discretion to determine whether two successful methods are meaningfully distinct.
• Did your honeypot successfully categorize calls based on the likelihood that the calls are robocalls?  Contestants will receive up to 30 points based on the percentage of calls that are successfully categorized. Contestants will need to prove to the Judges’ satisfaction that the calls were accurately categorized, and the necessary level of proof is within the Judge’s sole discretion.
• Is your honeypot scalable?  For each distinct method that your honeypot uses to categorize or verify the accuracy of the data collected, Contestants will receive five points for each such method that is easily replicable and adaptable. Furthermore, the Judges have sole discretion to determine whether two methods are meaningfully distinct.
• The Judges will compare the total points earned by each Contestant in this category and use these totals to assign a percentage amount of no greater than 70% to each Contestant.

 (ii) Explaining the Scheme (20% of total score)

• What insights did your Submission demonstrate with respect to setting up an effective robocall honeypot?
• What insights did your Submission demonstrate in determining the accuracy of the data captured?
• What insights did your Submission demonstrate in determining how to categorize calls based on the likelihood that the calls are robocalls?

(iii) Innovation (10% of total score)

• How innovative was your Submission?

B.  In order to be considered for a Prize, Submissions must score at least one percentage point in each required category (building knowledge, explaining the scheme, and innovation). If the Judges determine that no one satisfies each required category, no one will be deemed eligible for any Prize.

C.  The one (1) Contestant whose Submission earns the highest overall score will be named Winner of the phase 1 Top Prize identified below in Section 10, if the Contestant satisfies the verification requirements described in Section 11. If the Contestant does not satisfy the verification requirements, the phase 1 Top Prize may be awarded to the next highest scorer who satisfies the verification requirements, at the Judges’ or Sponsor’s discretion. 

D.  Up to two (2) Contestants with the subsequent highest scores who meet the Section 11 verification requirements may be awarded the phase 1 Honorable Mention Prizes—described below in Section 10—at the Judges’ or Sponsor’s discretion. 

E.  In the event of a tie between or among two or more Submissions where the Contestants meet the verification requirements, the relevant Prize identified below in Section 10 will be divided equally between the tied Contestants.

Phase 2: Attacker

A. Phase 2 Contestants will look into the minds of robocallers. Each Contestant will receive a list of 25 phone numbers that belong to a robocall honeypot set up on the Twilio platform. Contestants will also have free access to a Twilio account with $15 of credit that may be applied toward Twilio features. Contestants will attempt to circumvent the robocall honeypot. Merely spoofing the caller ID information (i.e., providing inaccurate or missing Caller ID data) will not be counted as circumvention of the robocall honeypot. One example of a way that Contestants can successfully circumvent the robocall honeypot is by placing undetected calls to the robocall honeypot numbers. A call is “undetected” if it prevents the honeypot from gathering any data about the call, including that the call was made. Contestants will be disqualified if they: 1) place calls using an autodialing program or the functional equivalent; 2) adversely affect the robocall honeypot; 3) adversely affect the Twilio platform; 4) adversely affect any other platform, product, system, or technology; 5) violate the terms of service of Twilio or any other third party provider; 6) do anything prohibited by law; or 7) place calls to any number outside the robocall honeypot.  Contestants may use any tool or platform to circumvent the honeypot. For each circumvention effort, Contestants must provide the necessary means for Judges to test and replicate the attack. If Contestants use a tool other than Twilio to place calls that circumvent the honeypot, Contestants must also provide access to their call detail records or call logs. In addition, each Contestant will submit all source code and a written description consisting of fewer than 500 words summarizing: 1) the Contestant’s circumvention techniques and outcomes; 2) theoretical circumvention techniques that the Contestant did not test but would be useful to a robocaller; and 3) how the Contestant would build a better honeypot to prevent such techniques.

Judging Criteria:

(i) Hitting the Target (50% of total score)

• Did you succeed in circumventing the robocall honeypot? You will receive one point for each distinct method that you used to circumvent the robocall honeypot. Contestants will need to prove to the Judges’ satisfaction that they circumvented the honeypot, and the necessary level of proof is within the Judges’ sole discretion. Furthermore, the Judges have sole discretion to determine whether two circumvention methods are meaningfully distinct.
• The Judges will compare the total points earned by each Contestant in this category and use these totals to assign a percentage amount of no greater than 50% to each Contestant.

 (ii) Explaining the Scheme (20% of total score)

• What insights did your Submission demonstrate about how attackers might circumvent a robocall honeypot, including by placing undetected calls?
• What theoretical techniques did your Submission describe for how attackers might circumvent a robocall honeypot?

(iii) Rebuilding (10% of total score)

• What insights did your Submission demonstrate about building a better robocall honeypot?

(iv) Innovation (20% of total score)

• How innovative were your actual or proposed methods of circumventing the honeypot?

B. In order to be considered for a Prize, Submissions must score at least one percentage point in each required category (hitting the target, explaining the scheme, rebuilding, and innovation).  If the Judges determine that no one satisfies each required category, no one will be deemed eligible for any Prize.

C.  The one (1) Contestant whose Submission earns the highest overall score will be named Winner of the phase 2 Top Prize identified below in Section 10, if the Contestant satisfies the verification requirements described in Section 11. If the Contestant does not satisfy the verification requirements, the phase 2 Top Prize may be awarded to the next highest scorer who satisfies the verification requirements, at the Judges’ or Sponsor’s discretion. 

D.  Up to two (2) Contestants with the subsequent highest scores who meet the Section 11 verification requirements may be awarded the phase 2 Honorable Mention Prizes—described below in Section 10—at the Judges’ or Sponsor’s discretion. 

E.  In the event of a tie between or among two or more Submissions where the Contestants meet the verification requirements, the relevant Prize identified below in Section 10 will be divided equally between the tied Contestants. 

Phase 3: Detective

A. Each phase 3 Contestant will receive two sets of call data from an existing robocall honeypot. The Sponsor will provide this data at the FTC’s “Zapping Rachel” booth at DEF CON 22, beginning at 9:00 am (PDT) on August 7, 2014. The first data set will identify calls that, based on real-world information, are likely to have been a robocall (a call delivering a prerecorded message). Based on information provided in the first data set, Contestants will develop an algorithm and will predict which of the calls in the second data set are likely to be robocalls. In addition to submitting these predictions, each Contestant will submit all source code and a written description of the algorithm consisting of fewer than 250 words.

Judging Criteria:

(i) Uncovering the Truth (70% of total score)

• Did you correctly predict whether the calls in the second honeypot data set were likely to be robocalls? To assess this, the Judges will compare your predictions with real-world information about which calls are likely to be a robocall. You will receive one point for each call you successfully identified as a likely robocall, and deducted one point for each call you inaccurately identified as a likely robocall.
• The Judges will compare the total points earned by each Contestant in this category and use these totals to assign a percentage amount of no greater than 70% to each Contestant.

(ii) Explaining the Scheme (20% of total score)

• What insights did your Submission demonstrate with respect to the analysis of honeypot call records?

(iii) Innovation (10% of total score)

• How innovative was your Submission?

B.  In order to be considered for a Prize, Submissions must score at least one percentage point in each required category (uncovering the truth, explaining the scheme, and innovation). If the Judges determine that no one satisfies each required category, no one will be deemed eligible for any Prize.

C.  The one (1) Contestant whose Submission earns the highest overall score will be named Winner of the phase 3 Top Prize identified below in Section 10, if the Contestant satisfies the verification requirements described in Section 11. If the Contestant does not satisfy the verification requirements, the phase 3 Top Prize may be awarded to the next highest scorer who satisfies the verification requirements, at the Judges’ or Sponsor’s discretion. 

D.  Up to two (2) Contestants with the subsequent highest scores who meet the Section 11 verification requirements may be awarded the phase 3 Honorable Mention Prizes—described below in Section 10—at the Judges’ or Sponsor’s discretion. 

E.  In the event of a tie between or among two or more Submissions where the Contestants meet the verification requirements, the relevant Prize identified below in Section 10 will be divided equally between the tied Contestants.