Tags


Problems and solutions with Gitea 1.15

First written on September 13, 2021
Last updated on October 29, 2021

My Gitea instance at software.franco.net.eu.org has been on SQLite since its inception in October 2018, up until some weeks ago. I never had any problems concerning database updates. Then I decided to migrate to something more scalable: MariaDB.

Migration of Gitea < 1.15.x from SQLite to MariaDB

Migration was made possible using the provided dump tool. Here is what I did (these instructions have been written after some trials and errors):

  1. create a MySQL user

    mysql -u root
    
    CREATE USER 'gitea' IDENTIFIED BY '${DB_PASSWORD}';
    
  2. create a MySQL database

    CREATE DATABASE giteadb CHARACTER SET 'utf8mb4' COLLATE 'utf8mb4_unicode_ci';
    GRANT ALL PRIVILEGES ON giteadb.* TO 'gitea';
    FLUSH PRIVILEGES;
    exit
    
  3. Go into the site administration and delete all authentication sources. This issue also happened to me.
  4. Stop Gitea

    systemctl stop gitea
    
  5. disable all cron jobs at gitea boot. Gitea didn’t start while cron jobs were enabled at boot. Edit /etc/gitea/app.ini:

    [cron]
    ; Enable running cron tasks periodically.
    ENABLED      = true
    ; Run cron tasks when Gitea starts.
    RUN_AT_START = false
    
  6. start and stop Gitea to have a clean restart without cron jobs running:

    systemctl start gitea
    sleep 60
    systemctl stop gitea
    
  7. create a database dump translated from sqlite to mysql. This command creates a zip file called gitea-dump-${id}.zip:

    sudo -i -u gitea
    HOME=/var/lib/gitea GITEA_WORK_DIR=/var/lib/gitea /usr/bin/gitea dump --skip-repository --skip-log --skip-custom-dir --skip-lfs-data --skip-attachment-data --database mysql -c /etc/gitea/app.ini
    
  8. change the database type in /etc/gitea/app.ini. Comment out the old section and add a new mysql section like this:

    [database]
    ; Either "mysql", "postgres", "mssql" or "sqlite3", it's your choice
    DB_TYPE             = mysql
    HOST                = 127.0.0.1:3306
    NAME                = giteadb
    USER                = gitea
    ; Use PASSWD = `your password` for quoting if you use special characters in the password.
    PASSWD              = `${DB_PASSWORD}`
    
  9. restore the database:

    unzip gitea-dump-${id}.zip
    mysql --default-character-set=utf8mb4 -ugitea -p${DB_PASSWORD} giteadb <gitea-db.sql
    
  10. systemctl start gitea
  11. re-add the authentication sources

This worked although some warnings remained when I lauched Gitea’s doctor command.

Update to 1.15.2

After updating the packages, Gitea did not start. There were a bunch of errors (for example this)! I was able to recover the database by hand by having a look at the errors from the log.

After some time, however, I noticed that the mirror feature did not work and the repositories on the web UI did not update after pushing from git.

Jump to the solution

Revert

I tried reverting back to a previous database version but once you migrate you cannot go back easily.

PostgreSQL

I tried with a fresh PostgreSQL 13 database and Gitea 1.15.2. The repostitory and mirror problem was still there so at least I knew it wasn’t a problem with the database.

What I did in the end was to re-install Gitea 1.14.7 and to switch to another new PostgreSQL database instead. The problem now was that I needed to recover most “issues”, mirrors, users, etc… from the original MariaDB database. Once that was done after hard work I made a backup and proceeded to the migration of this new database to Gitea 1.15.0.

Steps

These steps work on Debian GNU/Linux 10 which is now oldstable.

1. setup the database
  1. install PostgreSQL 13 from the official website
  2. Follow the database preparations instructions
  3. use the UNIX socket instead of TCP. This configuration might improve performance a little bit. Set this in /etc/gitea/app.ini

    [database]
    DB_TYPE             = postgres
    HOST                = /var/run/postgresql
    LOG_SQL             = false
    
2. recover the repositories

Create the users and push or “adopt” the repositories through the admin/repos URL. You will see the Unadopted Repository button there.

3. Adminer

Adminer is similar to phpMyAdmin. You can install it with:

apt-get install adminer/buster-backports

The backports version seem to be fully compatible with PostgreSQL 13.

Apache configuration
  1. if you use PHP-FPM you can use this snippet in a VirtualHost directive:

    Alias /dbadmin /usr/share/adminer/adminer/
    <Directory "/usr/share/adminer/adminer/">
        Require ip 127.0.0.1
        Require ip 192.168.0.
        Options FollowSymlinks
    </Directory>
    Include conf-enabled/php7.3-fpm.conf
    
  2. add this to the php.ini file of Apache, in /etc/php/7.3/apache2/php.ini:

    [HOST=my.host]
    open_basedir = /tmp/:/usr/share/adminer:/usr/share/php:/var/log/adminer
    
  3. finally:

    systemctl restart apache2
    
  4. connect to http://my.host/dbadmin

4. set the packages on hold

Set these packages on hold so when you update the system they don’t get updated by accident.

apt-mark hold gitea linux-image-amd64 postgresql-13 postgresql-client-13 postgresql-client-common postgresql-common

This is what you should get when you run apt-mark showhold:

gitea
linux-image-amd64
postgresql-13
postgresql-client-13
postgresql-client-common
postgresql-common
5. restore issues and comments
  1. open the original database with adminer
  2. get the repository id. Use the repository table to get it. Let’s say it’s 38
  3. go to the issue table
  4. click on the search filter on the top. Set repo_id = 38. Then click Select
  5. on the web-ui with the new database create new issues within the repository. In our example we have 9 issues so we need to create 9 dummy issues. Comment content and the user creating them is irrelevant
  6. now take note of the issues ids. In our case they are [1-9]
  7. open the comment table and do a new query, filtering by issue id. Let’s start with 1 In this case we only have 1 comment so we must create a dummy comment in the issue on web UI
  8. for issue 2 we have 2 comments so we need to create 2 dummy comments
  9. do this for issues [1-9]
  10. once we have all the dummy data we can open the new database on adminer. What we have to do now is just to manually copy-paste all the data from one database to the other. Check the UI once in a while if everything is working
Gitea issue table on Adminer
Gitea issue table on Adminer
Gitea comment table on Adminer
Gitea comment table on Adminer
Gitea comment table for issue 2 on Adminer
Gitea comment table for issue 2 on Adminer
6. restore mirrors from bare repositories
  1. once you have the repositories go into the repositories table
  2. select the to-be mirrors repositories and set the is_mirror variable to true
  3. go into the mirror table and create a new element. Use the correct repo_id.
Mirror variable in a repository
Mirror variable in a repository
An example row of the mirror table
An example row of the mirror table
7. restore users

To restore users simply create new users using Gitea’s admin interface and then copy-paste the data in the rows.

8. Database backups

I now use 15 minute separated backups using borgmatic. See also https://docs.franco.net.eu.org/automated-tasks/scripts.html#borgmatic-hooks-py

This is just an example for reference:

#
# borgmatic.iron_postgresql_giteadb.yaml
#
# Copyright (C) 2014-2020 Dan Helfman <https://torsion.org/borgmatic/docs/reference/config.yaml>
#               2021 Franco Masotti
#
# This program is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program.  If not, see <https://www.gnu.org/licenses/>.

location:
    source_directories: []
    repositories:
        - user@remotepc:backups/postgresql_giteadb.borg
storage:
    checkpoint_interval: 900
    lock_wait: 120
retention:
    keep_within: 1w
    keep_monthly: 1
consistency:
    checks:
        - archives
output:
    color: false
hooks:
    after_everything:
        - /home/jobs/scripts/by-user/root/borgmatic_hooks.py /home/jobs/scripts/by-user/root/borgmatic_hooks.iron_postgresql_giteadb.yaml 'finish' "{configuration_filename}" "{repository}" "{output}" "{error}"
    on_error:
        - /home/jobs/scripts/by-user/root/borgmatic_hooks.py /home/jobs/scripts/by-user/root/borgmatic_hooks.iron_postgresql_giteadb.yaml 'error' "{configuration_filename}" "{repository}" "{output}" "{error}"

    postgresql_databases:
        - name: giteadb
          # Use unix sockets instead of TCP.
          # See
          # https://torsion.org/borgmatic/docs/reference/configuration/
          #
          # hostname: 127.0.0.1
          # port: 5432
          username: gitea
          password: ${DB_PASS}
          format: tar
          options: "--verbose"

Extras

Torification

This is useful if you need to clone repositories, or do mirroring via TOR. I did this before switching to PostgreSQL:

  1. Install TOR and configure it

    apt-get install tor
    
  2. run systemctl edit gitea.service and add this content:

    [Unit]
    Description=Gitea (Git with a cup of tea)
    After=syslog.target
    After=network.target
    After=mysqld.service
    After=postgresql.service
    After=memcached.service
    After=redis.service
    
    # Comment or change these.
    Requires=network.target
    Requires=postgresql.service
    Requires=redis.service
    
    # Comment this if you don't need it.
    # See
    # https://docs.franco.net.eu.org/automated-tasks/scripts.html#notify-unit-status-py
    OnFailure=notify-unit-status@%n.service
    
    [Service]
    ExecStart=
    ExecStart=/usr/bin/torsocks --isolate /usr/bin/gitea web -c /etc/gitea/app.ini
    
    User=gitea
    Group=gitea
    Type=simple
    WorkingDirectory=~
    RuntimeDirectory=gitea
    LogsDirectory=gitea
    StateDirectory=gitea
    Environment=USER=gitea HOME=/var/lib/gitea GITEA_WORK_DIR=/var/lib/gitea
    Restart=always
    RestartSec=2s
    CapabilityBoundingSet=
    NoNewPrivileges=false
    #SecureBits=noroot-locked
    ProtectSystem=strict
    ProtectHome=true
    ReadWritePaths=/etc/gitea/app.ini
    PrivateTmp=true
    PrivateDevices=false
    PrivateUsers=false
    ProtectHostname=false
    ProtectClock=false
    ProtectKernelTunables=false
    ProtectKernelModules=false
    ProtectKernelLogs=false
    ProtectControlGroups=true
    LockPersonality=false
    MemoryDenyWriteExecute=false
    RestrictRealtime=false
    RestrictSUIDSGID=false
    SystemCallArchitectures=
    SystemCallFilter=
    SystemCallErrorNumber=EPERM
    ReadWriteDirectories=/var/spool/postfix/maildrop
    
  3. restart gitea

    systemctl restart gitea.servvice
    
  4. Test by cloning a repository from The Tor Project. Mirror this repository for example:

    http://eweiibe6tdjsdprb4px6rqrzzcsi22m4koia44kc5pcjr7nec2rlxyad.onion/tpo/anti-censorship/bridgedb.git

HTTP2

  1. enable HTTP2 on Apache

    a2enmod http2
    
  2. add this to your Apache configuration file. This setting can be global or per-virtualhost:

    Protocols h2 h2c http/1.1
    
  3. if you have problems with HTTP2 add this to /etc/gitea/app.ini:

    [ui.notification]
    EVENT_SOURCE_UPDATE_TIME=-1
    

    See also https://github.com/go-gitea/gitea/issues/11978

Caching

Solution

The repository/mirror problem was still there. After lots of trials and errors I found a solution.

Redis
  1. install Redis

    apt-get install redis-server
    
  2. setup Redis to listen on a UNIX socket exclusively. Edit /etc/redis/redis.conf:

    unixsocket /var/run/redis/redis-server.sock
    unixsocketperm 770
    bind 127.0.0.1 ::1
    
    # This disables listening on TCP.
    port 0
    
    # Avoid saving on disk
    save ""
    
    # Remove authentication.
    requirepass ""
    
  3. add the gitea user to the redis group so Gitea can have access to the socket.

    usermod -aG redis gitea
    
  4. restart Redis and Gitea:

    systemctl restart redis-server.service gitea.service
    
  5. check the logs at /var/log/redis/redis-server.log. If you have a warning about transparent hugepages. Add this service to Systemd.

    # See
    # https://unix.stackexchange.com/a/363887
    # https://stackoverflow.com/a/64945381
    # CC BY-SA 3.0
    # (c) 2017 nelaaro
    
    [Unit]
    Description=madvise Transparent Huge Pages
    Before=redis-server.service
    Before=apache2.service
    Before=gitea.service
    Before=postgresql.service
    
    [Service]
    Type=oneshot
    ExecStart=/bin/sh -c "/usr/bin/echo "madvise" | tee /sys/kernel/mm/transparent_hugepage/enabled"
    ExecStart=/bin/sh -c "/usr/bin/echo "madvise" | tee /sys/kernel/mm/transparent_hugepage/defrag"
    
    [Install]
    WantedBy=multi-user.target
    
  6. have a look at the Redis ArchWiki page.

Gitea configuration

To be able to use Redis in Gitea edit these sections in /etc/gitea/app.ini:

[cache]
ADAPTER  = redis
HOST     =  network=unix,addr=/var/run/redis/redis-server.sock,db=0,pool_size=100,idle_timeout=180s
ITEM_TTL = 24h

[session]
PROVIDER          = redis
PROVIDER_CONFIG  =  network=unix,addr=/var/run/redis/redis-server.sock,db=1,pool_size=100,idle_timeout=180s

[queue]
TYPE = redis
CONN_STR = network=unix,addr=/var/run/redis/redis-server.sock,db=2,pool_size=100,idle_timeout=180s

[queue.task]
QUEUE_TYPE = redis
QUEUE_CONN_STR = network=unix,addr=/var/run/redis/redis-server.sock,db=2,pool_size=100,idle_timeout=180s

[task]
QUEUE_TYPE = redis
QUEUE_CONN_STR = network=unix,addr=/var/run/redis/redis-server.sock,db=2,pool_size=100,idle_timeout=180s

Note: using Redis for queues and tasks seems to have solved the original problem.

Updating gitea next time

  1. temporarly disable the service

    systemctl stop gitea.service
    systemctl mask gitea.service
    
  2. copy the original database into a new one

    sudo -i -u postgres
    psql
    
    CREATE DATABASE backup_giteadb_${major}_${minor}_${patch} WITH TEMPLATE giteadb;
    

    where major, minor and patch correspond to the current Gitea version.

  3. run the doctor on the original database

    exit
    exit
    sudo -i -u gitea
    HOME=/var/lib/gitea GITEA_WORK_DIR=/var/lib/gitea /usr/bin/gitea -c /etc/gitea/app.ini doctor --all
    HOME=/var/lib/gitea GITEA_WORK_DIR=/var/lib/gitea /usr/bin/gitea -c /etc/gitea/app.ini doctor --all --fix
    
  4. update Gitea

     exit
     sudo -i
     apt-mark unhold gitea
     apt-get update
     apt-get dist-upgrade
    
  5. migrate the database

    exit
    sudo -i -u gitea
    HOME=/var/lib/gitea GITEA_WORK_DIR=/var/lib/gitea /usr/bin/gitea -c /etc/gitea/app.ini migrate
    
  6. run the doctor again

    HOME=/var/lib/gitea GITEA_WORK_DIR=/var/lib/gitea /usr/bin/gitea -c /etc/gitea/app.ini doctor --all
    HOME=/var/lib/gitea GITEA_WORK_DIR=/var/lib/gitea /usr/bin/gitea -c /etc/gitea/app.ini doctor --all --fix
    
  7. re-enable the service

    exit
    sudo -i
    systemctl unmask gitea.service
    systemctl start gitea.service
    apt-mark hold gitea
    

Apache2 reverse proxy for Gitea

  1. modify Gitea configuration (/etc/gitea/app.ini):

    [server]
    PROTOCOL                   = unix
    DOMAIN                     = localhost
    HTTP_ADDR                   = /var/run/gitea/gitea.sock
    UNIX_SOCKET_PERMISSION      = 770
    
    ; Do not set this variable if PROTOCOL is set to 'unix'.
    # LOCAL_ROOT_URL             = %(PROTOCOL)s://%(HTTP_ADDR)s:%(HTTP_PORT)s/
    
  2. Add the www-data user to gitea’s group so it can access the socket.

    usermod -aG gitea www-data
    
  3. Restart gitea and apache

    systemctl restart gitea.service apache2.service
    
  4. clearnet configuration in /etc/apache2/apache2.conf or a virtual host file:

    variables (shell style):

    • ONION_ADDRESS: the TOR address where Gitea is exposed
    • TCP_PORT: the TCP/IP port where gitea should be listenting. Even if we are using a UNIX socket this is applicable. You can use Gitea’s default port.
    • SERVER_NAME: the FQDN
    #########
    # Gitea #
    #########
    <IfModule mod_ssl.c>
    <VirtualHost *:443>
        UseCanonicalName on
    
        ProxyPreserveHost On
    
        Keepalive On
        RewriteEngine on
        AllowEncodedSlashes NoDecode
    
        ServerName ${SERVER_NAME}
    
        ProxyBadHeader Ignore
    
        # TOR.
        # Remove this if you don't want Gitea being avaliable on TOR.
        ServerAlias ${ONION_ADDRESS}
        Header set Onion-Location "http://${ONION_ADDRESS}%{REQUEST_URI}s"
    
        SSLCompression      off
    
        ProxyPass  / unix:/var/run/gitea/gitea.sock|http://127.0.0.1:${TCP_PORT}/ nocanon
        ProxyPassReverse  / unix:/var/run/gitea/gitea.sock|http://127.0.0.1:${TCP_PORT}/
    
        Include /etc/letsencrypt/options-ssl-apache.conf
        SSLCertificateFile /etc/letsencrypt/live/${SERVER_NAME}/fullchain.pem
        SSLCertificateKeyFile /etc/letsencrypt/live/${SERVER_NAME}/privkey.pem
    </VirtualHost>
    
  5. if you use TOR add this configuration in /etc/apache2/apache2.conf or a virtual host file:

    variables (shell style):

    • TCP_PORT: the TCP/IP port where Gitea should be listenting. Even if we are using a UNIX socket this is applicable. You can use Gitea’s default port
    • APACHE_TCP_PORT: the TCP/IP port where Apache is listening
    • SERVER_NAME: TOR’s FQDN
    #########
    # Gitea #
    #########
    <IfModule mod_ssl.c>
    <VirtualHost *:${APACHE_TCP_PORT}>
       UseCanonicalName on
    
       ProxyPreserveHost On
       ProxyRequests off
       AllowEncodedSlashes NoDecode
    
       Keepalive On
    
       RewriteEngine on
       ServerName ${SERVER_NAME}
    
       SSLCompression      off
    
       # Disable HTTP push and cloning for TOR.
       # TODO: enable only cloning but not push.
       RewriteRule  ^(.*)/info/refs /errors
       <Location "/errors">
           Deny from all
       </Location>
    
       # Redirect api and login to black holes.
       # Specific rules first, generic rules last.
       ProxyPass  /user/login unix:/var/run/gitea/gitea.sock|http://127.0.0.1:${TCP_PORT}/404 nocanon
       ProxyPassReverse  /user/login/ unix:/var/run/gitea/gitea.sock|http://127.0.0.1:${TCP_PORT}/404
       ProxyPass  /api unix:/var/run/gitea/gitea.sock|http://127.0.0.1:${TCP_PORT}/404 nocanon
       ProxyPassReverse  /api unix:/var/run/gitea/gitea.sock|http://127.0.0.1:${TCP_PORT}/404
       ProxyPass  / unix:/var/run/gitea/gitea.sock|http://127.0.0.1:${TCP_PORT}/ nocanon
       ProxyPassReverse  / unix:/var/run/gitea/gitea.sock|http://127.0.0.1:${TCP_PORT}/
       RequestHeader set X-Forwarded-Proto "http"
       RequestHeader set X-Forwarded-Port "${APACHE_TCP_PORT}"
       RequestHeader set X-Forwarded-Host "${SERVER_NAME}"
       # Distinguish normal traffic from tor's.
       RequestHeader set X-Forwarded-For "tor"
    </VirtualHost>
    </IfModule>
    
  6. if you use TOR, add this to /etc/tor/torrc

    variables (shell style):

    • APACHE_TCP_PORT: the TCP/IP port where Apache is listening
    HiddenServiceDir /var/lib/tor/gitea/
    HiddenServicePort 80 127.0.0.1:${APACHE_TCP_PORT}
    

    Follow these instructions.

See these links for the UNIX socket explanation:

Updates

After updating from 1.15.2 to 1.15.6, when running

HOME=/var/lib/gitea GITEA_WORK_DIR=/var/lib/gitea /usr/bin/gitea -c /etc/gitea/app.ini doctor --all --fix

I get this:

[4] Check consistency of database
 - [C] Error: pq: syntax error at or near "." whilst counting Collaborations without existing user

GitHub issues


Post a new comment