Enlarge (credit: Benj Edwards / Getty Images)

On Tuesday, researchers from Stanford University and University of California, Berkeley published a research paper that purports to show changes in GPT-4 's outputs over time. The paper fuels a common-but-unproven belief that the AI language model has grown worse at coding and compositional tasks over the past few months. Some experts aren't convinced by the results, but they say that the lack of certainty points to a larger problem with how OpenAI handles its model releases.

In a study titled "How Is ChatGPT’s Behavior Changing over Time?" published on arXiv, Lingjiao Chen, Matei Zaharia, and James Zou, cast doubt on the consistent performance of OpenAI's large language models (LLMs), specifically GPT-3.5 and GPT-4. Using API access , they tested the March and June 2023 versions of these models on tasks like math problem-solving, answering sensitive questions, code generation, and visual reasoning. Most notably, GPT-4's ability to identify prime numbers reportedly plunged dramatically from an accuracy of 97.6 percent in March to just 2.4 percent in June. Strangely, GPT-3.5 showed improved performance in the same period.

Performance of the March 2023 and June 2023 versions of GPT-4 and GPT-3.5 on four tasks, taken from "How Is ChatGPT’s Behavior Changing over Time?" (credit: Chen/Zaharia/Zou)

This study comes on the heels of people frequently complaining that GPT-4 has subjectively declined in performance over the past few months. Popular theories about why include OpenAI "distilling" models to reduce their computational overhead in a quest to speed up the output and save GPU resources, fine-tuning (additional training) to reduce harmful outputs that may have unintended effects, and a smattering of unsupported conspiracy theories such as OpenAI reducing GPT-4's coding capabilities so more people will pay for GitHub Copilot.

Enlarge (credit: Benj Edwards / Getty Images)

On Tuesday, researchers from Stanford University and University of California, Berkeley published a research paper that purports to show changes in GPT-4 's outputs over time. The paper fuels a common-but-unproven belief that the AI language model has grown worse at coding and compositional tasks over the past few months. Some experts aren't convinced by the results, but they say that the lack of certainty points to a larger problem with how OpenAI handles its model releases.

In a study titled "How Is ChatGPT’s Behavior Changing over Time?" published on arXiv, Lingjiao Chen, Matei Zaharia, and James Zou, cast doubt on the consistent performance of OpenAI's large language models (LLMs), specifically GPT-3.5 and GPT-4. Using API access , they tested the March and June 2023 versions of these models on tasks like math problem-solving, answering sensitive questions, code generation, and visual reasoning. Most notably, GPT-4's ability to identify prime numbers reportedly plunged dramatically from an accuracy of 97.6 percent in March to just 2.4 percent in June. Strangely, GPT-3.5 showed improved performance in the same period.

Performance of the March 2023 and June 2023 versions of GPT-4 and GPT-3.5 on four tasks, taken from "How Is ChatGPT’s Behavior Changing over Time?" (credit: Chen/Zaharia/Zou)

This study comes on the heels of people frequently complaining that GPT-4 has subjectively declined in performance over the past few months. Popular theories about why include OpenAI "distilling" models to reduce their computational overhead in a quest to speed up the output and save GPU resources, fine-tuning (additional training) to reduce harmful outputs that may have unintended effects, and a smattering of unsupported conspiracy theories such as OpenAI reducing GPT-4's coding capabilities so more people will pay for GitHub Copilot.

Enlarge (credit: Benj Edwards / Getty Images)

On Tuesday, researchers from Stanford University and University of California, Berkeley published a research paper that purports to show changes in GPT-4 's outputs over time. The paper fuels a common-but-unproven belief that the AI language model has grown worse at coding and compositional tasks over the past few months. Some experts aren't convinced by the results, but they say that the lack of certainty points to a larger problem with how OpenAI handles its model releases.

In a study titled "How Is ChatGPT’s Behavior Changing over Time?" published on arXiv, Lingjiao Chen, Matei Zaharia, and James Zou, cast doubt on the consistent performance of OpenAI's large language models (LLMs), specifically GPT-3.5 and GPT-4. Using API access , they tested the March and June 2023 versions of these models on tasks like math problem-solving, answering sensitive questions, code generation, and visual reasoning. Most notably, GPT-4's ability to identify prime numbers reportedly plunged dramatically from an accuracy of 97.6 percent in March to just 2.4 percent in June. Strangely, GPT-3.5 showed improved performance in the same period.

Performance of the March 2023 and June 2023 versions of GPT-4 and GPT-3.5 on four tasks, taken from "How Is ChatGPT’s Behavior Changing over Time?" (credit: Chen/Zaharia/Zou)

This study comes on the heels of people frequently complaining that GPT-4 has subjectively declined in performance over the past few months. Popular theories about why include OpenAI "distilling" models to reduce their computational overhead in a quest to speed up the output and save GPU resources, fine-tuning (additional training) to reduce harmful outputs that may have unintended effects, and a smattering of unsupported conspiracy theories such as OpenAI reducing GPT-4's coding capabilities so more people will pay for GitHub Copilot.

Enlarge (credit: Sean Gallup | Getty Images )

The Internet is dangerous, so what if you just didn't use it? That's the somewhat ironic recommendation Google, one of the world's largest Internet companies, is making to its employees. CNBC's Jennifer Elias reports that Google is "starting a new pilot program where some employees will be restricted to Internet-free desktop PCs" while they work. An internal memo seen by CNBC notes that “Googlers are frequent targets of attacks” by criminals, and a great way to combat that is to not be on the Internet.

Employees that work at major tech companies are a much richer target for criminals compared to normal people. Tech company employees have all sorts of access to sensitive data, and compromising a single employee could lead to exploiting sensitive infrastructure. Just last week, Microsoft was targeted by a Chinese espionage hacking group that somehow stole a cryptographic key to bypass Microsoft's authentication systems, giving it access to 25 organizations, including multiple government agencies.

The report says Google's new pilot program "will disable Internet access on the select desktops, with the exception of internal web-based tools and Google-owned websites like Google Drive and Gmail." This was originally mandatory for the 2,500 employees that were selected, but after "receiving feedback"—we're going to assume that was very enthusiastic feedback—Google is letting employees opt out of the program. The company also wants some employees to work without root access, which is common sense for a lot of computer roles, but not really for developers, which are used to being able to install new programs and tools.

Enlarge (credit: Sean Gallup | Getty Images )

The Internet is dangerous, so what if you just didn't use it? That's the somewhat ironic recommendation Google, one of the world's largest Internet companies, is making to its employees. CNBC's Jennifer Elias reports that Google is "starting a new pilot program where some employees will be restricted to Internet-free desktop PCs" while they work. An internal memo seen by CNBC notes that “Googlers are frequent targets of attacks” by criminals, and a great way to combat that is to not be on the Internet.

Employees that work at major tech companies are a much richer target for criminals compared to normal people. Tech company employees have all sorts of access to sensitive data, and compromising a single employee could lead to exploiting sensitive infrastructure. Just last week, Microsoft was targeted by a Chinese espionage hacking group that somehow stole a cryptographic key to bypass Microsoft's authentication systems, giving it access to 25 organizations, including multiple government agencies.

The report says Google's new pilot program "will disable Internet access on the select desktops, with the exception of internal web-based tools and Google-owned websites like Google Drive and Gmail." This was originally mandatory for the 2,500 employees that were selected, but after "receiving feedback"—we're going to assume that was very enthusiastic feedback—Google is letting employees opt out of the program. The company also wants some employees to work without root access, which is common sense for a lot of computer roles, but not really for developers, which are used to being able to install new programs and tools.

Enlarge (credit: Sean Gallup | Getty Images )

The Internet is dangerous, so what if you just didn't use it? That's the somewhat ironic recommendation Google, one of the world's largest Internet companies, is making to its employees. CNBC's Jennifer Elias reports that Google is "starting a new pilot program where some employees will be restricted to Internet-free desktop PCs" while they work. An internal memo seen by CNBC notes that “Googlers are frequent targets of attacks” by criminals, and a great way to combat that is to not be on the Internet.

Employees that work at major tech companies are a much richer target for criminals compared to normal people. Tech company employees have all sorts of access to sensitive data, and compromising a single employee could lead to exploiting sensitive infrastructure. Just last week, Microsoft was targeted by a Chinese espionage hacking group that somehow stole a cryptographic key to bypass Microsoft's authentication systems, giving it access to 25 organizations, including multiple government agencies.

The report says Google's new pilot program "will disable Internet access on the select desktops, with the exception of internal web-based tools and Google-owned websites like Google Drive and Gmail." This was originally mandatory for the 2,500 employees that were selected, but after "receiving feedback"—we're going to assume that was very enthusiastic feedback—Google is letting employees opt out of the program. The company also wants some employees to work without root access, which is common sense for a lot of computer roles, but not really for developers, which are used to being able to install new programs and tools.

An Anopheles stephensi mosquito, which can carry the malaria parasite. (credit: CDC)

A seventh person has been diagnosed with a locally acquired case of malaria in Florida's Sarasota County, state health officials reported this week .

The rare outbreak is now in its third month after authorities in the Sunshine State reported the first case in May. When Florida had identified four cases by late June, the Centers for Disease Control and Prevention released a health alert to clinicians calling locally acquired malaria cases in the US a "public health emergency."

Florida's outbreak and a single, unrelated case in Texas from June collectively mark the first time in two decades that the US has seen locally acquired malaria cases, which, if left untreated, can be deadly. In the last instance, in 2003, Florida officials reported a small outbreak of at least seven people in Palm Beach.

An Anopheles stephensi mosquito, which can carry the malaria parasite. (credit: CDC)

A seventh person has been diagnosed with a locally acquired case of malaria in Florida's Sarasota County, state health officials reported this week .

The rare outbreak is now in its third month after authorities in the Sunshine State reported the first case in May. When Florida had identified four cases by late June, the Centers for Disease Control and Prevention released a health alert to clinicians calling locally acquired malaria cases in the US a "public health emergency."

Florida's outbreak and a single, unrelated case in Texas from June collectively mark the first time in two decades that the US has seen locally acquired malaria cases, which, if left untreated, can be deadly. In the last instance, in 2003, Florida officials reported a small outbreak of at least seven people in Palm Beach.

An Anopheles stephensi mosquito, which can carry the malaria parasite. (credit: CDC)

A seventh person has been diagnosed with a locally acquired case of malaria in Florida's Sarasota County, state health officials reported this week .

The rare outbreak is now in its third month after authorities in the Sunshine State reported the first case in May. When Florida had identified four cases by late June, the Centers for Disease Control and Prevention released a health alert to clinicians calling locally acquired malaria cases in the US a "public health emergency."

Florida's outbreak and a single, unrelated case in Texas from June collectively mark the first time in two decades that the US has seen locally acquired malaria cases, which, if left untreated, can be deadly. In the last instance, in 2003, Florida officials reported a small outbreak of at least seven people in Palm Beach.