La captura de datos es el proceso de identificar y capturar cambios de datos en la base de datos.

Con captura de datos, los cambios en los datos pueden ser rastreados casi en tiempo real, y esa información puede ser utilizada para apoyar una variedad de casos de uso, incluyendo auditoría, replicación y sincronización.

Un buen ejemplo de un caso de uso para captura de datos es considerar una aplicación que inserta un registro en la base de datos y envía un evento a una cola de mensajes después de que se ha insertado el registro (escribir dos veces).

Imagina que estás trabajando en una aplicación de comercio electrónico y después de que se crea y se inserta un pedido en la base de datos, se envía un evento OrderCreated a una cola de mensajes. Los consumidores del evento podrían hacer cosas como crear órdenes de recolección para el almacén, programar transportes para la entrega y enviar un correo electrónico de confirmación del pedido al cliente.

Pero ¿qué sucede si la aplicación se bloquea después de que se ha insertado el pedido en la base de datos pero antes de lograr enviar el evento a la cola de mensajes? Esto es posible debido al hecho de que no se puede insertar atómicamente el registro Y enviar el mensaje en la misma transacción, por lo que si la aplicación se bloquea después de insertar el registro en la base de datos pero antes de enviar el evento a la cola, se pierde el evento.

Por supuesto, existen soluciones alternativas para evitar esto: una solución simple es “almacenar” el evento en una tabla de almacenamiento temporal en la misma transacción en la que se escribe el registro, y luego depender de un proceso captura de datos para capturar el cambio en la tabla de almacenamiento y enviar el evento a la cola de mensajes. La transacción es atómica y el proceso de captura de datos puede asegurar que el evento se entregue al menos una vez.

Para capturar cambios, la captura de datos típicamente utiliza uno de dos métodos: basado en registro o basado en disparadores.

La captura de datos basado en registro implica leer los registros de transacciones de la base de datos para identificar los cambios de datos, que es el método que utilizaremos aquí al utilizar la replicación lógica de Postgres.

Replicación de Postgres

Hay dos modos de replicación en Postgres:

1. Replicación física: cada cambio del primario se transmite a las réplicas a través del WAL (Write Ahead Log). Esta replicación se realiza byte por byte con direcciones de bloque exactas.

1. Replicación lógica: en la replicación lógica, el suscriptor recibe cada cambio de transacción individual (es decir, declaraciones INSERT, UPDATE o DELETE) en la base de datos.

El WAL todavía se transmite, pero codifica las operaciones lógicas para que puedan ser decodificadas por el suscriptor sin tener que conocer los detalles internos de Postgres.

Una de las grandes ventajas de la replicación lógica es que se puede utilizar para replicar sólo tablas o filas específicas, lo que significa que se tiene un control completo sobre lo que se está replicando.

Para habilitar la replicación lógica, el wal_level debe ser configurado:

-- determines how much information is written to the wal. 
-- Each 'level' inherits the level below it; 'logical' is the highest level

ALTER SYSTEM SET wal_level=logical;

-- simultaneously running WAL sender processes
ALTER SYSTEM SET max_wal_senders='10';

-- simultaneously defined replication slots
ALTER SYSTEM SET max_replication_slots='10';

Los cambios requieren un reinicio de la instancia de Postgres.

Después de reiniciar el sistema, el wal_level se puede verificar con:

SHOW wal_level;
 wal_level 
-----------
 logical
(1 row)

Para suscribirse a los cambios se debe crear una publicación . Una publicación es un grupo de tablas en las que nos gustaría recibir cambios de datos.

Vamos a crear una tabla simple y definir una publicación para ella:

CREATE TABLE articles (id serial PRIMARY KEY, title text, description text, body text);
CREATE PUBLICATION articles_pub FOR TABLE articles;

Para indicar a Postgres que retenga segmentos de WAL, debemos crear un slot de replicación.

El slot de replicación representa un flujo de cambios desde una o más publicaciones y se utiliza para prevenir la pérdida de datos en caso de una falla del servidor, ya que son a prueba de fallos.

Protocolo de Replicación

Para tener una idea del protocolo y los mensajes que se envían, podemos usar pg_recvlogical para iniciar un suscriptor de replicación:

# Start and use the publication defined above
# output is written to stdout
pg_recvlogical --start \
  --host='localhost' \
  --port='5432' \
  --username='postgres' \
  --dbname='postgres' \
  --option=publication_names='articles_pub' \
  --option=proto_version=1 \
  --create-slot \
  --if-not-exists \
  --slot=articles_slot \
  --plugin=pgoutput \
  --file=-

Insertar un registro:

INSERT INTO articles (title, description, body)
    VALUES ('Postgres replication', 'Using logical replication', 'Foo bar baz');

Cada linea en la salida corresponde a un mensaje de replicación recibido a través de suscripción:

B(egin) - Begin transaction 
R(elation) - Table, schema, columns and their types
I(insert) - Data being inserted
C(ommit) - Commit transaction

___________________________________

B

Rarticlesdidtitledescriptionbody
It35tPostgres replicationtUsing logical replicationtFoo bar baz
C

Si insertamos múltiples registros en una transacción deberíamos tener dos I entre B y C:

BEGIN;
INSERT INTO articles (title, description, body) VALUES ('First', 'desc', 'Foo');

INSERT INTO articles (title, description, body) VALUES ('Second', 'desc', 'Bar');
COMMIT;

Y la salida:

C
B

It37tFirsttdesctFoo
It38tSecondtdesctBar
CCopied to clipboard!

La información de la relación, es decir, la tabla, no se transmitió porque ya se recibió la relación al insertar el primer registro.

Postgres solo envía la relación la primera vez que se encuentra durante la sesión. Se espera que el suscriptor almacene en caché una relación previamente enviada.

Ahora que tenemos una idea de cómo funciona la replicación lógica, ¡implementémosla en Elixir!

Implementando la conexión de replicación

Cree un nuevo proyecto de Elixir:

mix new cdc

Añadiremos las siguientes dependencias a mix.exs:

defp deps do
  {:postgrex, "~> 0.16.4"},
  # decode/encode replication messages
  {:postgrex_pgoutput, "~> 0.1.0"}
end

Postgrex admite la replicación a través del proceso Postgrex.ReplicationConnection.

defmodule CDC.Replication do
  use Postgrex.ReplicationConnection
  require Logger

  defstruct [
    :publications,
    :slot,
    :state
  ]

  def start_link(opts) do
    conn_opts = [auto_reconnect: true]
    publications = opts[:publications] || raise ArgumentError, message: "`:publications` is required"
    slot = opts[:slot] || raise ArgumentError, message: "`:slot` is required"

    Postgrex.ReplicationConnection.start_link(
      __MODULE__,
      {slot, publications},
      conn_opts ++ opts
    )
  end

  @impl true
  def init({slot, pubs}) do
    {:ok, %__MODULE__{slot: slot, publications: pubs}}
  end

  
  @impl true
  def handle_connect(%__MODULE__{slot: slot} = state) do
    query = "CREATE_REPLICATION_SLOT #{slot} TEMPORARY LOGICAL pgoutput NOEXPORT_SNAPSHOT"

    Logger.debug("[create slot] query=#{query}")

    {:query, query, %{state | state: :create_slot}}
  end

  @impl true
  def handle_result([%Postgrex.Result{} | _], %__MODULE__{state: :create_slot, publications: pubs, slot: slot} = state) do
    opts = [proto_version: 1, publication_names: pubs]

    query = "START_REPLICATION SLOT #{slot} LOGICAL 0/0 #{escape_options(opts)}"

    Logger.debug("[start streaming] query=#{query}")

    {:stream, query, [], %{state | state: :streaming}}
  end

  @impl true
  def handle_data(msg, state) do
    Logger.debug("Received msg=#{inspect(msg, limit: :infinity, pretty: true)}")
    {:noreply, [], state}
  end

  defp escape_options([]),
    do: ""

  defp escape_options(opts) do
    parts =
      Enum.map_intersperse(opts, ", ", fn {k, v} -> [Atom.to_string(k), ?\s, escape_string(v)] end)

    [?\s, ?(, parts, ?)]
  end

  defp escape_string(value) do
    [?', :binary.replace(to_string(value), "'", "''", [:global]), ?']
  end
end

El código esta disponible en GitHub

Probemos:

opts = [
  slot: "articles_slot_elixir",
  publications: ["articles_pub"],
  host: "localhost",
  database: "postgres",
  username: "postgres",
  password: "postgres",
  port: 5432,
]

CDC.Replication.start_link(opts)

Cuando iniciamos el proceso, ocurre lo siguiente:

1. Una vez que estamos conectados a Postgres, se llama al callback handle_connect/1, se crea un slot de replicación lógica temporal.
2. Se llama a handle_result/2 con el resultado de la consulta en el paso 1. Si el slot se creó correctamente, comenzamos a transmitir desde el slot y entramos en el modo de transmisión. La posición solicitada ‘0/0’ significa que Postgres elige la posición.
3. Cualquier mensaje de replicación enviado desde Postgres se recibe en el callback handle_data/2.

Mensajes de replicación

Hay dos tipos de mensajes que un suscriptor recibe:

1. primary_keep_alive: un mensaje de comprobación, si reply == 1 se espera que el suscriptor responda al mensaje con un standby_status_update para evitar una desconexión por tiempo de espera.

El standby_status_update contiene el LSN actual que el suscriptor ha procesado.

Postgres utiliza este mensaje para determinar qué segmentos de WAL se pueden eliminar de forma segura.

1. xlog_data: contiene los mensajes de datos para cada paso en una transacción.Dado que no estamos respondiendo a los mensajes primary_keep_alive, el proceso se desconecta y se reinicia.

Arreglemos esto decodificando los mensajes y comenzando a responder con mensajes standby_status_update.

defmodule CDC.Replication do
  use Postgrex.ReplicationConnection

  require Postgrex.PgOutput.Messages
  alias Postgrex.PgOutput.{Messages, Lsn}

  require Logger

  defstruct [
    :publications,
    :slot,
    :state
  ]

  def start_link(opts) do
    conn_opts = [auto_reconnect: true]
    publications = opts[:publications] || raise ArgumentError, message: "`:publications` is required"
    slot = opts[:slot] || raise ArgumentError, message: "`:slot` is required"

    Postgrex.ReplicationConnection.start_link(
      __MODULE__,
      {slot, publications},
      conn_opts ++ opts
    )
  end

  @impl true
  def init({slot, pubs}) do
    {:ok, %__MODULE__{slot: slot, publications: pubs}}
  end

  @impl true
  def handle_connect(%__MODULE__{slot: slot} = state) do
    query = "CREATE_REPLICATION_SLOT #{slot} TEMPORARY LOGICAL pgoutput NOEXPORT_SNAPSHOT"

    Logger.debug("[create slot] query=#{query}")

    {:query, query, %{state | state: :create_slot}}
  end

  @impl true
  def handle_result(
        [%Postgrex.Result{} | _],
        %__MODULE__{state: :create_slot, publications: pubs, slot: slot} = state
      ) do
    opts = [proto_version: 1, publication_names: pubs]

    query = "START_REPLICATION SLOT #{slot} LOGICAL 0/0 #{escape_options(opts)}"

    Logger.debug("[start streaming] query=#{query}")

    {:stream, query, [], %{state | state: :streaming}}
  end

  @impl true
  def handle_data(msg, state) do
    return_msgs =
      msg
      |> Messages.decode()
      |> handle_msg()

    {:noreply, return_msgs, state}
  end

  #
  defp handle_msg(Messages.msg_primary_keep_alive(server_wal: lsn, reply: 1)) do
    Logger.debug("msg_primary_keep_alive message reply=true")
    <<lsn::64>> = Lsn.encode(lsn)

    [standby_status_update(lsn)]
  end

  defp handle_msg(Messages.msg_primary_keep_alive(reply: 0)), do: []

  defp handle_msg(Messages.msg_xlog_data(data: data)) do
    Logger.debug("xlog_data message: #{inspect(data, pretty: true)}")
    []
  end

  defp standby_status_update(lsn) do
    [
      wal_recv: lsn + 1,
      wal_flush: lsn + 1,
      wal_apply: lsn + 1,
      system_clock: Messages.now(),
      reply: 0
    ]
    |> Messages.msg_standby_status_update()
    |> Messages.encode()
  end

  
defp escape_options([]),
    do: ""

  defp escape_options(opts) do
    parts =
      Enum.map_intersperse(opts, ", ", fn {k, v} -> [Atom.to_string(k), ?\s, escape_string(v)] end)

    [?\s, ?(, parts, ?)]
  end

  defp escape_string(value) do
    [?', :binary.replace(to_string(value), "'", "''", [:global]), ?']
  end
end

handle_data/2 decodifica el mensaje y lo pasa a handle_msg/1. Si es un primary_keep_alive , respondemos con un standby_status_update.

El LSN denota una posición de byte en el WAL.

El suscriptor responde con el LSN que ha manejado actualmente, como no estamos haciendo seguimiento de los mensajes que recibimos, simplemente confirmamos con el LSN enviado desde el servidor.

A continuación, manejaremos los mensajes xlog_data, la idea aquí es que capturaremos cada operación en una estructura de transacción.

Capturando transacciones

El módulo CDC.Protocol manejará los mensajes xlog_data y rastreará el estado de la transacción

defmodule CDC.Protocol do
  import Postgrex.PgOutput.Messages
  require Logger

  alias CDC.Tx
  alias Postgrex.PgOutput.Lsn

  
@type t :: %__MODULE__{
          tx: Tx.t(),
          relations: map()
        }

  defstruct [
    :tx,
    relations: %{}
  ]

  @spec new() :: t()
  def new do
    %__MODULE__{}
  end

  def handle_message(msg, state) when is_binary(msg) do
    msg
    |> decode()
    |> handle_message(state)
  end

  def handle_message(msg_primary_keep_alive(reply: 0), state), do: {[], nil, state}
  def handle_message(msg_primary_keep_alive(server_wal: lsn, reply: 1), state) do
    Logger.debug("msg_primary_keep_alive message reply=true")
    <<lsn::64>> = Lsn.encode(lsn)

    {[standby_status_update(lsn)], nil, state}
  end

  def handle_message(msg, %__MODULE__{tx: nil, relations: relations} = state) do
    tx =
      [relations: relations, decode: true]
      |> Tx.new()
      |> Tx.build(msg)

    {[], nil, %{state | tx: tx}}
  end

  def handle_message(msg, %__MODULE__{tx: tx} = state) do
    case Tx.build(tx, msg) do
      %Tx{state: :commit, relations: relations} ->
        tx = Tx.finalize(tx)
        relations = Map.merge(state.relations, relations)
        {[], tx, %{state | tx: nil, relations: relations}}

      tx ->
        {[], nil, %{state | tx: tx}}
    end
  end

  defp standby_status_update(lsn) do
    [
      wal_recv: lsn + 1,
      wal_flush: lsn + 1,
      wal_apply: lsn + 1,
      system_clock: now(),
      reply: 0
    ]
    |> msg_standby_status_update()
    |> encode()
  end
end

CDC.Tx maneja mensajes recibidos dentro de la transacción, begin, relation, insert/update/delete y commit.

defmodule CDC.Tx do
  import Postgrex.PgOutput.Messages
  alias Postgrex.PgOutput.Lsn

  alias __MODULE__.Operation

  @type t :: %__MODULE__{
          operations: [Operation.t()],
          relations: map(),
          timestamp: term(),
          xid: pos_integer(),
          state: :begin | :commit,
          lsn: Lsn.t(),
          end_lsn: Lsn.t()
        }

  defstruct [
    :timestamp,
    :xid,
    :lsn,
    :end_lsn,
    relations: %{},
    operations: [],
    state: :begin,
    decode: true
  ]

  def new(opts \\ []) do
    struct(__MODULE__, opts)
  end

  def finalize(%__MODULE__{state: :commit, operations: ops} = tx) do
    %{tx | operations: Enum.reverse(ops)}
  end

  def finalize(%__MODULE__{} = tx), do: tx

  @spec build(t(), tuple()) :: t()
  def build(tx, msg_xlog_data(data: data)) do
    build(tx, data)
  end

  def build(tx, msg_begin(lsn: lsn, timestamp: ts, xid: xid)) do
    %{tx | lsn: lsn, timestamp: ts, xid: xid, state: :begin}
  end

  def build(%__MODULE__{state: :begin, relations: relations} = tx, msg_relation(id: id) = rel) do
    %{tx | relations: Map.put(relations, id, rel)}
  end

  def build(%__MODULE__{state: :begin, lsn: tx_lsn} = tx, msg_commit(lsn: lsn, end_lsn: end_lsn))
      when tx_lsn == lsn do
    %{tx | state: :commit, end_lsn: end_lsn}
  end

  def build(%__MODULE__{state: :begin} = builder, msg_insert(relation_id: id) = msg),
    do: build_op(builder, id, msg)

  def build(%__MODULE__{state: :begin} = builder, msg_update(relation_id: id) = msg),
    do: build_op(builder, id, msg)

  def build(%__MODULE__{state: :begin} = builder, msg_delete(relation_id: id) = msg),
    do: build_op(builder, id, msg)

  # skip unknown messages
  def build(%__MODULE__{} = tx, _msg), do: tx

  defp build_op(%__MODULE__{state: :begin, relations: rels, decode: decode} = tx, id, msg) do
    rel = Map.fetch!(rels, id)
    op = Operation.from_msg(msg, rel, decode)

    %{tx | operations: [op | tx.operations]}
  end
end

CDC.Tx.Operation maneja los mensajes INSERT/UPDATE/DELETE y decodifica los datos combinándolos con la relación

defmodule CDC.Tx.Operation do
  @moduledoc "Describes a change (INSERT, UPDATE, DELETE) within a transaction."

  import Postgrex.PgOutput.Messages
  alias Postgrex.PgOutput.Type, as: PgType

  @type t :: %__MODULE__{}
  defstruct [
    :type,
    :schema,
    :namespace,
    :table,
    :record,
    :old_record,
    :timestamp
  ]

  @spec from_msg(tuple(), tuple(), decode :: boolean()) :: t()
  def from_msg(
        msg_insert(data: data),
        msg_relation(columns: columns, namespace: ns, name: name),
        decode?
      ) do
    %__MODULE__{
      type: :insert,
      namespace: ns,
      schema: into_schema(columns),
      table: name,
      record: cast(data, columns, decode?),
      old_record: %{}
    }
  end

  def from_msg(
        msg_update(change_data: data, old_data: old_data),
        msg_relation(columns: columns, namespace: ns, name: name),
        decode?
      ) do
    %__MODULE__{
      type: :update,
      namespace: ns,
      table: name,
      schema: into_schema(columns),
      record: cast(data, columns, decode?),
      old_record: cast(columns, old_data, decode?)
    }
  end

  def from_msg(
        msg_delete(old_data: data),
        msg_relation(columns: columns, namespace: ns, name: name),
        decode?
      ) do
    %__MODULE__{
      type: :delete,
      namespace: ns,
      schema: into_schema(columns),
      table: name,
      record: %{},
      old_record: cast(data, columns, decode?)
    }
  end

  defp into_schema(columns) do
    for c <- columns do
      c
      |> column()
      |> Enum.into(%{})
    end
  end

  defp cast(data, columns, decode?) do
    Enum.zip_reduce([data, columns], %{}, fn [text, typeinfo], acc ->
      key = column(typeinfo, :name)

      value =
        if decode? do
          t =
            typeinfo
            |> column(:type)
            |> PgType.type_info()

          PgType.decode(text, t)
        else
          text
        end

      Map.put(acc, key, value)
    end)
  end
end

Como antes, el mensaje primary_keep_alive con reply == 1 envía un standby_status_update. Cuando recibimos un mensaje xlog_data, creamos un nuevo %Tx{} que usamos para “construir” la transacción hasta que recibimos un msg_commit que marca el final de la transacción.

Cualquier mensaje de inserción, actualización o eliminación crea una CDC.Tx.Operation en la transacción, cada operación contiene un relation_id que se utiliza para buscar la relación desde tx.relations.

La operación junto con la relación nos permite decodificar los datos. La información de columna y tipo se recupera de la relación y se utiliza para decodificar los valores en términos de Elixir.

Una vez que estamos en un estado de commit, fusionamos Tx.relations con Protocol.relations, ya que un mensaje de relación sólo se transmitirá la primera vez que se encuentre una tabla durante la sesión de conexión, Protocol.relations contiene todos los msg_relation que se nos han enviado durante la sesión.

El módulo CDC.Replication ahora se ve así:

defmodule CDC.Replication do
  use Postgrex.ReplicationConnection

  alias CDC.Protocol

  require Logger

  defstruct [
    :publications,
    :protocol,
    :slot,
    :state
  ]

  def start_link(opts) do
    conn_opts = [auto_reconnect: true]
    publications = opts[:publications] || raise ArgumentError, message: "`:publications` is required"
    slot = opts[:slot] || raise ArgumentError, message: "`:slot` is required"

    Postgrex.ReplicationConnection.start_link(
      __MODULE__,
      {slot, publications},
      conn_opts ++ opts
    )
  end

  @impl true
  def init({slot, pubs}) do
    {:ok,
     %__MODULE__{
       slot: slot,
       publications: pubs,
       protocol: Protocol.new()
     }}
  end

  @impl true
  def handle_connect(%__MODULE__{slot: slot} = state) do
    query = "CREATE_REPLICATION_SLOT #{slot} TEMPORARY LOGICAL pgoutput NOEXPORT_SNAPSHOT"

    Logger.debug("[create slot] query=#{query}")

    {:query, query, %{state | state: :create_slot}}
  end

  @impl true
  def handle_result(
        [%Postgrex.Result{} | _],
        %__MODULE__{state: :create_slot, publications: pubs, slot: slot} = state
      ) do
    opts = [proto_version: 1, publication_names: pubs]

    query = "START_REPLICATION SLOT #{slot} LOGICAL 0/0 #{escape_options(opts)}"

    Logger.debug("[start streaming] query=#{query}")

    {:stream, query, [], %{state | state: :streaming}}
  end

  @impl true
  def handle_data(msg, state) do
    {return_msgs, tx, protocol} = Protocol.handle_message(msg, state.protocol)

    if not is_nil(tx) do
      Logger.debug("Tx: #{inspect(tx, pretty: true)}")
    end

    {:noreply, return_msgs, %{state | protocol: protocol}}
  end

  
defp escape_options([]),
    do: ""

  defp escape_options(opts) do
    parts =
      Enum.map_intersperse(opts, ", ", fn {k, v} -> [Atom.to_string(k), ?\s, escape_string(v)] end)

    [?\s, ?(, parts, ?)]
  end

  defp escape_string(value) do
    [?', :binary.replace(to_string(value), "'", "''", [:global]), ?']
  end
end

handle_data/2 llama a Protocol.handle_message/1 que devuelve una tupla con tres elementos {messages_to_send :: [binary()], complete_transaction :: CDC.Tx.t() | nil, CDC.Protocolo.t()}

Por ahora solo inspeccionamos la transacción cuando se emite desde Protocol.handle_message/3, probémoslo:

Interactive Elixir (1.14.0) - press Ctrl+C to exit (type h() ENTER for help)
opts = [
  slot: "articles_slot_elixir",
  publications: ["articles_pub"],
  host: "localhost",
  database: "postgres",
  username: "postgres",
  password: "postgres",
  port: 5432,
]

{:ok, _} = CDC.Replication.start_link(opts)
{:ok, pid} = Postgrex.start_link(opts)

insert_query = """
INSERT INTO articles (title, description, body) 
VALUES ('Postgres replication', 'Using logical replication', 'with Elixir!')
"""

_ = Postgrex.query!(pid, insert_query, [])
  
14:03:48.020 [debug] Tx: %CDC.Tx{
  timestamp: ~U[2022-10-31 13:03:48Z],
  xid: 494,
  lsn: {0, 22981920},
  end_lsn: nil,
  relations: %{
    16386 => {:msg_relation, 16386, "public", "articles", :default,
     [
       {:column, [:key], "id", :int4, -1},
       {:column, [], "title", :text, -1},
       {:column, [], "description", :text, -1},
       {:column, [], "body", :text, -1}
     ]}
  },
  operations: [
    %CDC.Tx.Operation{
      type: :insert,
      schema: [
        %{flags: [:key], modifier: -1, name: "id", type: :int4},
        %{flags: [], modifier: -1, name: "title", type: :text},
        %{flags: [], modifier: -1, name: "description", type: :text},
        %{flags: [], modifier: -1, name: "body", type: :text}
      ],
      namespace: "public",
      table: "articles",
      record: %{
        "body" => "with Elixir!",
        "description" => "Using logical replication",
        "id" => 6,
        "title" => "Postgres replication"
      },
      old_record: %{},
      timestamp: nil
    }
  ],
  state: :begin,
  decode: true
}

Cada cambio en la transacción se almacena en Tx.operations, operation.record es la fila decodificada como un mapa.

Finalmente, implementemos una forma de suscribirnos a los cambios de CDC.Replication:

defmodule CDC.Replication do
  use Postgrex.ReplicationConnection

  alias CDC.Protocol

  require Logger

  defstruct [
    :publications,
    :protocol,
    :slot,
    :state,
    subscribers: %{}
  ]

  def start_link(opts) do
    conn_opts = [auto_reconnect: true]
    publications = opts[:publications] || raise ArgumentError, message: "`:publications` is required"
    slot = opts[:slot] || raise ArgumentError, message: "`:slot` is required"

    Postgrex.ReplicationConnection.start_link(
      __MODULE__,
      {slot, publications},
      conn_opts ++ opts
    )
  end

  def subscribe(pid, opts \\ []) do
    Postgrex.ReplicationConnection.call(pid, :subscribe, Keyword.get(opts, :timeout, 5_000))
  end

  def unsubscribe(pid, ref, opts \\ []) do
    Postgrex.ReplicationConnection.call(
      pid,
      {:unsubscribe, ref},
      Keyword.get(opts, :timeout, 5_000)
    )
  end

  @impl true
  def init({slot, pubs}) do
    {:ok,
     %__MODULE__{
       slot: slot,
       publications: pubs,
       protocol: Protocol.new()
     }}
  end

  @impl true
  def handle_connect(%__MODULE__{slot: slot} = state) do
    query = "CREATE_REPLICATION_SLOT #{slot} TEMPORARY LOGICAL pgoutput NOEXPORT_SNAPSHOT"

    Logger.debug("[create slot] query=#{query}")

    {:query, query, %{state | state: :create_slot}}
  end

  @impl true
  def handle_result(
        [%Postgrex.Result{} | _],
        %__MODULE__{state: :create_slot, publications: pubs, slot: slot} = state
      ) do
    opts = [proto_version: 1, publication_names: pubs]

    query = "START_REPLICATION SLOT #{slot} LOGICAL 0/0 #{escape_options(opts)}"

    Logger.debug("[start streaming] query=#{query}")

    {:stream, query, [], %{state | state: :streaming}}
  end

  @impl true
  def handle_data(msg, state) do
    {return_msgs, tx, protocol} = Protocol.handle_message(msg, state.protocol)

    if not is_nil(tx) do
      notify(tx, state.subscribers)
    end

    {:noreply, return_msgs, %{state | protocol: protocol}}
  end

  # Replies must be sent using `reply/2`
  # https://hexdocs.pm/postgrex/Postgrex.ReplicationConnection.html#reply/2
  @impl true
  def handle_call(:subscribe, {pid, _} = from, state) do
    ref = Process.monitor(pid)

    state = put_in(state.subscribers[ref], pid)

    Postgrex.ReplicationConnection.reply(from, {:ok, ref})

    {:noreply, state}
  end

  def handle_call({:unsubscribe, ref}, from, state) do
    {reply, new_state} =
      case state.subscribers do
        %{^ref => _pid} ->
          Process.demonitor(ref, [:flush])

          {_, state} = pop_in(state.subscribers[ref])
          {:ok, state}

        _ ->
          {:error, state}
      end

    from && Postgrex.ReplicationConnection.reply(from, reply)

    {:noreply, new_state}
  end

  @impl true
  def handle_info({:DOWN, ref, :process, _, _}, state) do
    handle_call({:unsubscribe, ref}, nil, state)
  end

  defp notify(tx, subscribers) do
    for {ref, pid} <- subscribers do
      send(pid, {:notification, self(), ref, tx})
    end

    :ok
  end

  defp escape_options([]),
    do: ""

  defp escape_options(opts) do
    parts =
      Enum.map_intersperse(opts, ", ", fn {k, v} -> [Atom.to_string(k), ?\s, escape_string(v)] end)

    [?\s, ?(, parts, ?)]
  end

  defp escape_string(value) do
    [?', :binary.replace(to_string(value), "'", "''", [:global]), ?']
  end
end

Y lo podemos usar así:

opts = [
  slot: "articles_slot",
  publications: ["articles_pub"],
  host: "localhost",
  database: "postgres",
  username: "postgres",
  password: "postgres",
  port: 5432,
]

{:ok, pid} = CDC.Replication.start_link(opts)
{:ok, pg_pid} = Postgrex.start_link(opts)
{:ok, ref} = CDC.Replication.subscribe(pid)

insert_query = """
INSERT INTO articles (title, description, body) 
VALUES ('Postgres replication', 'Using logical replication', 'with Elixir!')
"""

_ = Postgrex.query!(pg_pid, insert_query, [])
flush()

{:notification, #PID<0.266.0>, #Reference<0.2499916608.3416784901.94813>,
 %CDC.Tx{
   timestamp: ~U[2022-10-31 13:26:35Z],
   xid: 495,
   lsn: {0, 22983536},
   end_lsn: nil,
   relations: %{
     16386 => {:msg_relation, 16386, "public", "articles", :default,
      [
        {:column, [:key], "id", :int4, -1},
        {:column, [], "title", :text, -1},
        {:column, [], "description", :text, -1},
        {:column, [], "body", :text, -1}
      ]}
   },
   operations: [
     %CDC.Tx.Operation{
       type: :insert,
       schema: [
         %{flags: [:key], modifier: -1, name: "id", type: :int4},
         %{flags: [], modifier: -1, name: "title", type: :text},
         %{flags: [], modifier: -1, name: "description", type: :text},
         %{flags: [], modifier: -1, name: "body", type: :text}
       ],
       namespace: "public",
       table: "articles",
       record: %{
         "body" => "with Elixir!",
         "description" => "Using logical replication",
         "id" => 7,
         "title" => "Postgres replication"
       },
       old_record: %{},
       timestamp: nil
     }
   ],
   state: :begin,
   decode: true
 }}

Conclusión

Si está buscando una manera de capturar cambios de su base de datos con cambios mínimos en su configuración existente, definitivamente vale la pena considerar Cambiar la captura de datos. Con Elixir y postgrex hemos implementado un mini Debezium en ~400 LOC. La fuente completa está disponible aquí .

Si necesita ayuda con la implementación de Elixir, nuestro equipo de expertos líder en el mundo siempre está aquí para ayudarlo. Contáctenos hoy para saber cómo podemos ayudarlo.

The post Captura de datos con Postgres y Elixir appeared first on Erlang Solutions .

Cobalt 1.3 depends on M-Vault 19.0 or subsequent versions

M-Vault Management Supporties

• M-Vault Bootstrap.   Enables operation in conjunction with M-Vault 19.0 to support headless bootstrap.
• Managing users in M-Vault groups, such as Directory Server Administrators  and Messaging Configuration Read/Write.  This enables Cobalt to control user and operator rights to access M-Vault.
• AD/LDAP passthrough support
  • Allow users (per domain) to support mandatory or partial passthrough
  • Set and validate passthrough entry for user
  • Identify users in passthrough server that might be added to domain

Messaging Management

• Profile Editor for supporting and managing M-Switch Profiler.
  • SIC Coverage UI. Provide full list of SICS, showing which addresses each one goes to.   This enables operator to ensure that all SICs are sensibly handled.
• File Transfer By Email capability is now managed by Cobalt, replacing capability previously in MConsole.
• For Organizations and Military DLs enable control manage capability functions:
  • Max Message Size
  • Max Line Length (for ACP 127 destinations)
  • Charset Restrictions (for ACP 127 destinations)
  • Allows/block attachments
• Option to show for a user which DLs the user is in, and give easy addition to other DLs.  This facilitates managing DL membership.

New Views

• Non-Human Users (Special Users).  Need to support accounts with passwords that are not humans.   For XMPP, Email or both.
• View for end users, rather than administrators.  User can:
  • Change password.
  • See all of own entry and modify  attributes.   The list of modifiable attributes can be configured.
  • See references to entry and email list membership.
• User Groups, to enable management of directory groups (Distinguished Names).

Cobalt Access Control

• New Cobalt roles, that can enable selective control of which users can access directory admin controls, and which users can set OAUTH rights and can add OAUTH Clients.
• Restrict Password set/change rights, so that only selected Cobalt administrators can do this.

Security Enhancements

• When deleting a user, remove the password.   This will make it safe for applications searching whole DIT as you can’t authenticate with a deleted user’s account.
• Security Clearance can be selected for any role or user, based on a configured catalogue.  This supports key M-Switch and Harrier feature to check clearances.

Miscellaneous

• When assigning a new email, search entire DIT for conflicts, not just Cobalt area.   This  helps SASL resilience
• Can add Photos to Routed UAs and Organizations.
• Check References on Delete. Cobalt has a “References” button on user/role form that displays all references of a user/role.  On deleting, references are deleted as well.
• Tool to check references to users in AD, so that when users in AD are deleted, dangling references can be picked up.
• Remove default domain concept
• On deletion of domain in Cobalt, give option to delete all the domain data
• Option to end all  cobalt logged in sessions of an operator, to allow an operator to logout from all browsers with a single action
• There is also an option for an operator with appropriate rights  to end sessions of another Cobalt operator.

Read this post in Spanish .

Let’s talk about Elixir!

Elixir is a functional programming language created by José Valim to build concurrent and scalable systems. It is defined as:

“a dynamic, functional language for building scalable and maintainable applications”.

https://elixir-lang.org/

Its first version was released in 2012. Since then, new features and improvements have been added until its current version . It is a relatively young programming language that has established itself quickly due to its nice syntax and short learning curve. Also, it is supported by a technology that has been working since the eighties in production systems, the BEAM.

Elixir runs on the BEAM, Erlang’s virtual machine.

In a later blog post, we’ll go into more detail about the virtual machine. For now, I want to mention a few features that make the BEAM a solid technology and a fantastic choice for system development. For example:

• It simultaneously supports millions of users and transactions.
• It has a mechanism to detect failures and define strategies that allow a system to recover from them.
• It has all the necessary elements to develop a system capable of operating without interruptions or, failing that, with the minimum of them.
• It allows system updates in real-time without stopping and in an “invisible” way for end users.

Elixir inherits all these properties from BEAM. Adding to it that-the language has a very nice syntax, it is easy to learn, there are many resources (blogs, podcasts, etc), and the community is very friendly. So creating a project from scratch to start practicing requires very little time and effort.

I was introduced to Elixir by chance in 2018. The project I learned it with made me suffer a bit because it was a different experience than what I was used to until then. But once I got into the rhythm, it became pretty enjoyable. Sometime last year, I wondered what would have happened if I hadn’t discovered it by coincidence.

Would Elixir have caught my attention on its own? Is it a recommended option for someone learning to program? Do you require previous experience?

So I posted a tweet asking other developers:

Here are some answers :

It is not easy to reach a definitive conclusion since choosing this language, as the first option will depend on the tastes and experiences of the individual. Some of its advantages are mentioned in the tweet replies, which motivated me to write this series.

Without further ado, I welcome you to the series Elixir, 7 steps to start your journey.

Throughout seven chapters, we’ll talk a bit about history, relevant technical aspects and delve into why Elixir has quickly gained popularity. I will also tell you a little about my experience in the projects in which I have participated.

What topics will it cover?

1. Erlang’s virtual machine, the BEAM
2. Understanding processes and concurrency
3. Libraries and frameworks
4. Testing and debugging
5. The Elixir Community
6. Functional Programming vs. Object-Oriented Programming
7. My first project with Elixir!

Who is this series for?

• People with no experience in any programming language looking for an option to explore.
• People with previous experience in other programming languages who want to experiment with Elixir.

Difficulty level: Beginner

Elixir official documentation:

• Elixir language: https://elixir-lang.org/
• Elixir releases: https://elixir-lang.org/blog/2022/09/01/elixir-v1-14-0-released/

In each chapter, I’ll share resources to dig deeper into the topics. You can find me on Twitter as @loreniuxmr to clarify questions or continue the conversation. You can also use the hashtags #MyElixirStatus and #Elixirlang  to find other Elixir developers.

The next post will be about Erlang’s virtual machine, the BEAM, and why it is so relevant when it comes to Elixir. See you!

The post Elixir, 7 steps to start your journey appeared first on Erlang Solutions .

¡Hablemos de Elixir!

Elixir es un lenguaje de programación funcional creado por José Valim para construi sistemas concurrentes y escalables.

“a dynamic, functional language for building scalable and maintainable applications”.

https://elixir-lang.org/

Su primera versión fue liberada en 2012. Desde entonces, se le han agregado nuevas funcionalidades y mejoras hasta llegar a su versión actual . Se trata de un lenguaje de programación relativamente joven, pero bien establecido y que ha ganado aceptación rápidamente gracias su sintaxis agradable y una curva de aprendizaje corta. Elixir está respaldado por una tecnología que ha estado en funcionamiento en sistemas reales desde los años ochenta, la BEAM.

Elixir corre sobre la máquina virtual de Erlang, la BEAM.

Más adelante entraremos en más detalles y hablaremos de la máquina virtual. Por ahora me gustaría mencionar algunas características que hacen de la BEAM una tecnología sólida y una gran opción para el desarrollo de sistemas. Por ejemplo:

• Soporta simultáneamente millones de usuarios y transacciones .
• Tiene un mecanismo para detectar fallos y te permite definir estrategias para recuperarse de ellos.
• Te brinda todos los elementos necesarios para desarrollar sistemas capaces de operar sin interrupciones o, en su defecto, con el mínimo de ellas.
• Permite hacer actualizaciones de un sistema en tiempo real sin detenerlo, y de manera “invisible” para los usuarios finales.

Elixir hereda todas estas propiedades de la BEAM. Y a eso le sumamos que el lenguaje tiene una sintaxis bastante agradable, es fácil de aprender, existen muchos recursos (blogs, podcasts, etc) y la comunidad es increíble. Así que crear un proyecto desde cero para practicar y aprender require de muy poco tiempo y esfuerzo.

Conocí Elixir por casualidad en 2018. El proyecto con el que lo aprendí me hizo sufrir un poco, porque se trataba de un lenguaje totalmente diferente a los que había conocido hasta ese entonces (orientados a objetos) pero mi experiencia una vez que descubrí todo lo que había se volvió muy disfrutable. En algún momento del año pasado me pregunté qué hubiera pasado de no haberlo descubierto por coincidencia.

¿Me hubiera llamado la atención en algún momento? ¿Es una opción recomendada para alguien que está aprendiendo programación? ¿Requiere experiencia previa?

Así que le pregunté en Twitter a otros desarrolladores :

A continuación algunas respuestas:

Es difícil llegar a una conclusión definitiva, pues escoger este lenguaje como primera opción o no dependerá de los gustos y la experiencia de cada quien. Algunas de las respuestas hablan acerca de las ventajas de este lenguaje de programación y fueron la razón que me motivo a escribir esta serie.

Así que sin más que decir, bienvenido a Elixir, 7 pasos para iniciar tu viaje.

A lo largo de siete capítulos hablaremos un poco acerca de su historia, aspectos técnicos relevantes y por qué ha ganado popularidad rápidamente. También te contaré cómo ha sido mi experiencia en los proyectos en los que he participado.

¿Qué temas cubrirá?

1. La máquina virtual de Erlang, la BEAM
2. Entendiendo procesos y concurrencia
3. Bibliotecas y frameworks
4. Pruebas y debugging
5. La comunidad de Elixir
6. Programación funcional vs Programación orientada a objetos
7. ¡Mi primer proyecto con Elixir!

¿A quién está dirigida esta serie?

• Personas sin experiencia en ningún lenguaje de programación que están en busca de su primera opción para explorar.
• Personas con experiencia previa en otros lenguajes de programación que quieran experimentar con Elixir.

Nivel de dificultad: Principantes.

Documentación oficial de Elixir:

• Elixir language: https://elixir-lang.org/
• Elixir releases: https://elixir-lang.org/blog/2022/09/01/elixir-v1-14-0-released/

En cada capítulo te compartiré recursos para profundizar los temas. Puedes encontrarme en Twitter como @loreniuxmr para aclarar cualquier duda o seguir con la conversación. Y también puedes utilizar los siguientes hashtags para conectar con otros desarrolladores: #MyElixirStatus y #Elixirlang

En la próxima nota hablaremos de la máquina virtual de Erlang, la BEAM y por qué es tan relevante cuando hablamos de Elixir. ¡Nos vemos!

The post Elixir, 7 pasos para iniciar tu viaje appeared first on Erlang Solutions .

Dino is a secure and open-source messaging application. It uses the XMPP (Jabber) protocol for decentralized communication. We aim to provide an intuitive and enjoyable user interface.

The 0.4 release adds support for message reactions and replies. We also switched from GTK3 to GTK4 and make use of libadwaita now.

Reactions and Replies

Reactions give you a quick and light-weight way to respond to a message with an emoji. They can be used to express agreement or disagreement 👍️👎️, for voting, to express a feeling 🥳🤯, and much more 🦕. You can react with any emoji you want and with as many as you want!

The new Dino release also adds another way for you to interact with messages: Replies. The original message is displayed alongside the reply, and you can click on it to jump up to the original message.

You can add a reaction or reply to a message via the message menu. To access the message menu, hover the message or tap it on touch-screens.

Reactions and replies are always possible in direct conversations. In group chats and channels, the admin decides whether to support the features.

GTK4 and libadwaita

Dino now uses GTK4 instead of GTK3 to display its user interface. To the outside there are no big UI changes. However, we can now make use of new or improved GTK features.

Furthermore, we started using libadwaita, which contains specialized Widgets and provides tools to build mobile-friendly UIs. We already adjusted Dino’s main view for usage on mobile devices.

Ilulissat

Glaciers are fascinating landscapes of flowing ice. We named this Dino release “Ilulissat” after a glacier in Greenland to help spread information on the effects of global warming on glaciers.

Glaciers are created over a span of centuries from fallen snow that compacts and transforms into glacial ice. When the ice reaches a certain thickness, it starts to behave like a liquid. Thus, glaciers are in constant movement, driven downhill by gravity under their own weight.

Very large glaciers, also known as ice sheets, exist on Antarctica and Greenland. Greenland’s ice sheet covers about 80% of the island and has an average thickness of 1,5 km.

The Greenland Ice Sheet, like all glaciers, is constantly in motion. It accumulates ice in the interior of the island and flows outwards, eventually reaching the ocean through so-called outlet glaciers. One such outlet glacier is the Ilulissat glacier in West Greenland, the fastest draining outlet of the Greenland Ice Sheet with a flow speed of over 20 meters per day.

Unfortunately, the Greenland Ice Sheet has been shrinking for past 26 years [ 1 ]. Rising water and air temperatures are causing outlet glaciers to melt at an accelerating pace, draining the ice sheet more quickly and resulting in increased sea levels [ 2 ]. Between 1992 and 2020, meltwater from the Greenland Ice Sheet alone increased global sea levels by over 1,3 cm, where every centimeter of sea level rise is estimated to expose 6 Million people to coastal flooding [ 3 ].

We are thrilled to announce the latest update to our STANAG 5066 server, Icon-5066. With this new release, we’ve incorporated a host of exciting features and enhancements, designed to not only add new functionality to your deployment but also increase the performance of your HF Radio Network.

The below is a list of the changes, and updates that can be found within Icon-5066 v3.0.

ALE Management

This major new feature enables management of ALE configuration independent of ALE implementation and allows easy sharing of configuration between nodes.  This capability is supported for modems where Isode provides ALE support.  Key features:

• Web configuration of HF Network for each Icon-5066 node.
• Configuration of Node ALE addressing, with support for 2G, 3G and 4G.
• Support for fixed frequency (not using ALE for a network)
• Configuration of HF Frequency list with options for narrowband and wideband
• Configuration of schedules for used with ALE or fixed frequency.   This enables the frequencies used to be changed at configured times so that appropriate frequencies are used for an ALE network throughout the 24 hour cycle.
• Import/Export of configuration, to enable easy sharing of configuration between nodes.   Model is that you configure ALE setup on one node and the transfer to other nodes.

Security

Two important security enhancements are included:

1. Use of OAuth to control which operators can access Icon-5066.
2. Support of TLS which includes:
   1. HTTPS Web Access
   2. TLS Support for GCXP to support Modem Proxy (crypto bypass) across a Red/Black boundary
   3. Web configuration of PKI setup of TLS

STANAG 5066 Ed4 Compliance

Icon-5066 is compliant to STANAG 5066 Ed4.   An overview of Ed4 is here .   Detailed Icon-5066 compliance is specified here .

Most of these capabilities were in the previous release, but described as STANAG 5066 proposed extensions.   Interoperability has been tested with another Ed4 implementation.

SNR Monitor

A new option is provided to configure Icon-5066 as a modem monitor with a simple TCP monitoring protocol.  This is a general purpose capability, but is specifically targeted to support the ACP 127 FAB (Frequency Assignment Broadcast) capability in M-Switch to enable the FAB broadcast to report on measured link quality using a modem at a remote location.

New Modem/ALE Support

The following ALE capabilities are added:

• 3G ALE support for RapidM RM8 and RM10.
• 4G ALE support for RapidM RM10

A new “Raw TCP” data option, which sends and receives data over simple TCP connection.   This generic capability can be used to exchange data with RapidM RM10 modem.

Management

Support for independent control of multiple STANAG 5066 nodes, so that on a system with multiple nodes nodes can be independently enabled and disabled by the Icon-5066 operator.

Red/Black Driver

A driver is provided for Isode’s Red/Black product to monitor Icon-5066.   Like the Red/Black driver for Isode supported Modems, this driver is distributed with Icon-5066, but will be picked up by a collocated Red/Black server.   It enables a Red/Black operator to enable/disable an Icon-5066 node and to monitor key parameters.

Product Activation

Icon-5066 servers are now controlled by Isode Product Activation.  This control includes:

• Optional enabling of TLS.  This is helpful for export.
• Control of the number of nodes available

Almost three months after the previous release, ejabberd 23.01 includes many bug fixes, several improvements and some new features.

A new module, mod_mqtt_bridge , can be used to replicate changes to MQTT topics between local and remote servers.

A more detailed explanation of those topics and other features:

Erlang/OTP 19.3 discouraged

Remember that support for Erlang/OTP 19.3 is discouraged, and will be removed in a future release. Please upgrade to Erlang/OTP 20.0 or newer. Check more details in the ejabberd 22.10 release announcement .

New MQTT bridge

This new module allows synchronizing topic changes between local and remote servers. It can be configured to replicate local changes to remote server, or can subscribe to topics on remote server and update local copies when they change.

When connecting to a remote server you can use native or websocket encapsulated protocol, and you can connect using both v4 and v5 protocol. It can authenticate using username/password pairs or with client TLS certificates.

New Hooks

Regarding MQTT support, there are several new hooks:

• mqtt_publish : New hook for MQTT publish event
• mqtt_subscribe and mqtt_unsubscribe : New hooks for MQTT subscribe & unsubscribe events

New option log_modules_fully

The loglevel top-level option specifies the verbosity of log files generated by ejabberd.

If you want some specific modules to log everything, independently from whatever value you have configured in loglevel , now you can use the new log_modules_fully option.

For example, if you are investigating some problem in ejabberd_sm and mod_client_state :

loglevel: warning
log_modules_fully: [ejabberd_sm, mod_client_state]

(This option works only on systems with Erlang 22 or newer).

Changes in option outgoing_s2s_families

The outgoing_s2s_families top-level option specifies which address families to try, in what order.

The default value has now been changed to try IPv6 first, as servers are within data centers where IPv6 is more commonly enabled (contrary to clients). And if it’s not present, then it’ll just fall back to IPv4.

By the way, this option is obsolete and irrelevant when using ejabberd 23.01 and Erlang/OTP 22, or newer versions of them.

Changes in option captcha_cmd

The captcha_cmd top-level option specifies the full path to a script that can generate a CAPTCHA image. Now this option may specify an Erlang module name, which should implement a function to generate a CAPTCHA image.

ejabberd does not include any such module, but there are two available in the ejabberd-contrib repository that you can install and try: mod_ecaptcha and mod_captcha_rust .

DOAP file

The protocols implemented or supported by ejabberd are defined in the corresponding source code modules since ejabberd 15.06. Until now, only the XEP number and supported version were tracked. Since now, it’s possible to document what ejabberd version first implemented it, the implementation status and an arbitrary comment.

That information until now was only used by the script tools/check_xep_versions.sh . A new script is added, tools/generate-doap.sh , to generate a DOAP file with that information. A new target is added to Makefile: make doap .

And that DOAP file is now published as ejabberd.doap in the git repository. That file is read by the XMPP.org website to show ejabberd’s protocols, see XMPP Servers: ejabberd .

VSCode

Support for Visual Studio Code and variants is vastly improved. Thanks to the Erlang LS VSCode extension , the ejabberd git repository includes support for developing, compiling and debugging ejabberd with Visual Studio Code, VSCodium, Coder’s code-server and Github Codespaces.

See more details in the ejabberd Docs: VSCode page.

ChangeLog

General

• Add misc:uri_parse/2 to allow declaring default ports for protocols
• CAPTCHA: Add support to define module instead of path to script
• Clustering: Handle mnesia_system_event mnesia_up when other node joins this ( #3842 )
• ConverseJS: Don’t set i18n option because Converse enforces it instead of browser lang ( #3951 )
• ConverseJS: Try to redirect access to files mod_conversejs to CDN when there is no local copies
• ext_mod: compile C files and install them in ejabberd’s priv
• ext_mod: Support to get module status from Elixir modules
• make-binaries: reduce log output
• make-binaries: Bump zlib version to 1.2.13
• MUC: Don’t store mucsub presence events in offline storage
• MUC: hibernation_time is not an option worth storing in room state ( #3946 )
• Multicast: Jid format when multicastc was cached ( #3950 )
• mysql: Pass ssl options to mysql driver
• pgsql: Do not set standard_conforming_strings to off ( #3944 )
• OAuth: Accept jid as a HTTP URL query argument
• OAuth: Handle when client is not identified
• PubSub: Expose the pubsub#type field in disco#info query to the node ( #3914 )
• Translations: Update German translation

Admin

• api_permissions : Fix option crash when doesn’t have who: section
• log_modules_fully : New option to list modules that will log everything
• outgoing_s2s_families : Changed option’s default to IPv6, and fall back to IPv4
• Fix bash completion when using Relive or other install methods
• Fix portability issue with some shells ( #3970 )
• Allow admin command to subscribe new users to members_only rooms
• Use alternative split/2 function that works with Erlang/OTP as old as 19.3
• Silent warning in OTP24 about not specified cacerts in SQL connections
• Fix compilation warnings with Elixir 1.14

DOAP

• Support extended -protocol erlang attribute
• Add extended RFCs and XEP details to some protocol attributes
• tools/generate-doap.sh : New script to generate DOAP file, add make doap ( #3915 )
• ejabberd.doap : New DOAP file describing ejabberd supported protocols

MQTT

• Add MQTT bridge module
• Add support for certificate authentication in MQTT bridge
• Implement reload in MQTT bridge
• Add support for websockets to MQTT bridge
• Recognize ws5/wss5 urls in MQTT bridge
• mqtt_publish : New hook for MQTT publish event
• mqtt_(un)subscribe : New hooks for MQTT subscribe & unsubscribe events

VSCode

• Improve .devcontainer to use use devcontainer image and .vscode
• Add .vscode files to instruct VSCode how to run ejabberd
• Add Erlang LS default configuration
• Add Elvis default configuration

Full Changelog

https://github.com/processone/ejabberd/compare/22.10…23.01

ejabberd 23.01 download & feedback

As usual, the release is tagged in the Git source code repository on GitHub .

The source package and installers are available in ejabberd Downloads page. To check the *.asc signature files, see How to verify ProcessOne downloads integrity .

For convenience, there are alternative download locations like the ejabberd DEB/RPM Packages Repository and the GitHub Release / Tags .

The Docker image is in Docker Hub , and there’s an alternative Container image in GitHub Packages .

If you suspect that you’ve found a bug, please search or fill a bug report on GitHub Issues .

The post ejabberd 23.01 first appeared on ProcessOne .