ciphers/base32.py and ciphers/base85.py encode Unicode code points instead of bytes: output diverges from the spec for non-ASCII, round-trips corrupt chars >= U+0400, arbitrary bytes crash #15636

Repository commit

6e03a48

Python version (python --version)

Python 3.13.5

Dependencies version (pip freeze)

None needed — both modules only use the standard library.

Expected behavior

Base32 and Ascii85 are byte encodings: base32_encode(data) should return the RFC 4648 encoding of the input bytes (matching base64.b32encode), and ascii85_encode(data) should match base64.a85encode. They should accept arbitrary bytes, as the type hints (data: bytes) suggest. The sibling modules base16.py and base64_cipher.py do exactly this and match the standard library for every input.

Actual behavior

ciphers/base32.py and ciphers/base85.py start with data.decode("utf-8") and then encode Unicode code points (bin(ord(d))) instead of bytes. All three consequences show up as soon as the input is not pure ASCII:

1. Output diverges from the standard for non-ASCII text:

>>> from ciphers.base32 import base32_encode
>>> import base64
>>> base32_encode("héllo".encode())
b'NDUWY3DPA======='
>>> base64.b32encode("héllo".encode())
b'NDB2S3DMN4======'   # correct RFC 4648 result

2. Encode→decode silently corrupts data for characters ≥ U+0400 (their code point is wider than 8 bits, shifting every following bit):

>>> from ciphers.base32 import base32_encode, base32_decode
>>> base32_decode(base32_encode("€100".encode()))
b'\xc2\x82\xc2\xb0\xc3\x84\xc3\x80\xc3\x80'   # should round-trip to b'\xe2\x82\xac100'

Same for ascii85_decode(ascii85_encode("€100".encode())) → b'\xc2\x82\xc2\xb0\xc3\x84\xc3\x80\xc3\x80'.

3. Arbitrary non-UTF-8 bytes — perfectly valid input for a byte codec — raise UnicodeDecodeError:

>>> base32_encode(bytes([0xff, 0xfe, 0x01]))
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xff in position 0

The doctests only use ASCII strings, so the test suite never exercises any of this.

Steps to reproduce

python - <<'EOF'
import base64
from ciphers.base32 import base32_encode, base32_decode
from ciphers.base85 import ascii85_encode, ascii85_decode
for data in ["héllo".encode(), "€100".encode(), bytes([0xff, 0xfe, 0x01])]:
    print(data)
    try: print("  b32:", base32_encode(data), "| stdlib:", base64.b32encode(data))
    except Exception as e: print("  b32 crash:", repr(e))
    try: print("  b32 roundtrip ok:", base32_decode(base32_encode(data)) == data)
    except Exception as e: print("  b32 roundtrip crash:", repr(e))
    try: print("  a85 match:", ascii85_encode(data) == base64.a85encode(data))
    except Exception as e: print("  a85 crash:", repr(e))
EOF

Suspected cause

ciphers/base32.py:19 and ciphers/base85.py:25 both build the bit string from "".join(bin(ord(d))[2:].zfill(8) for d in data.decode("utf-8")). Operating on the raw bytes (bin(byte)[2:].zfill(8) for byte in data, like base16.py does) and assembling the decode side from byte values instead of chr(int(chunk, 2)) would keep both codecs on-spec and lossless for every input.