tt/AutoJs6
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

6.6.2 - Alpha5 - 本地化 Color Picker 库; 主题色设置页面新旧布局分离 (初步)

Browse Source
This commit is contained in:
SuperMonster003
2025-03-25 22:54:40 +08:00
parent 1b3ff3bb5a
commit 124dea0c0e
284 changed files with 6919 additions and 537 deletions
Download Patch File Download Diff File Expand all files Collapse all files

191
modules/jieba-analysis/LICENSE Normal file
View File

@@ -0,0 +1,191 @@
Apache License
Version 2.0, January 2004
http://www.apache.org/licenses/
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
1. Definitions.
"License" shall mean the terms and conditions for use, reproduction, and
distribution as defined by Sections 1 through 9 of this document.
"Licensor" shall mean the copyright owner or entity authorized by the copyright
owner that is granting the License.
"Legal Entity" shall mean the union of the acting entity and all other entities
that control, are controlled by, or are under common control with that entity.
For the purposes of this definition, "control" means (i) the power, direct or
indirect, to cause the direction or management of such entity, whether by
contract or otherwise, or (ii) ownership of fifty percent (50%) or more of the
outstanding shares, or (iii) beneficial ownership of such entity.
"You" (or "Your") shall mean an individual or Legal Entity exercising
permissions granted by this License.
"Source" form shall mean the preferred form for making modifications, including
but not limited to software source code, documentation source, and configuration
files.
"Object" form shall mean any form resulting from mechanical transformation or
translation of a Source form, including but not limited to compiled object code,
generated documentation, and conversions to other media types.
"Work" shall mean the work of authorship, whether in Source or Object form, made
available under the License, as indicated by a copyright notice that is included
in or attached to the work (an example is provided in the Appendix below).
"Derivative Works" shall mean any work, whether in Source or Object form, that
is based on (or derived from) the Work and for which the editorial revisions,
annotations, elaborations, or other modifications represent, as a whole, an
original work of authorship. For the purposes of this License, Derivative Works
shall not include works that remain separable from, or merely link (or bind by
name) to the interfaces of, the Work and Derivative Works thereof.
"Contribution" shall mean any work of authorship, including the original version
of the Work and any modifications or additions to that Work or Derivative Works
thereof, that is intentionally submitted to Licensor for inclusion in the Work
by the copyright owner or by an individual or Legal Entity authorized to submit
on behalf of the copyright owner. For the purposes of this definition,
"submitted" means any form of electronic, verbal, or written communication sent
to the Licensor or its representatives, including but not limited to
communication on electronic mailing lists, source code control systems, and
issue tracking systems that are managed by, or on behalf of, the Licensor for
the purpose of discussing and improving the Work, but excluding communication
that is conspicuously marked or otherwise designated in writing by the copyright
owner as "Not a Contribution."
"Contributor" shall mean Licensor and any individual or Legal Entity on behalf
of whom a Contribution has been received by Licensor and subsequently
incorporated within the Work.
2. Grant of Copyright License.
Subject to the terms and conditions of this License, each Contributor hereby
grants to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free,
irrevocable copyright license to reproduce, prepare Derivative Works of,
publicly display, publicly perform, sublicense, and distribute the Work and such
Derivative Works in Source or Object form.
3. Grant of Patent License.
Subject to the terms and conditions of this License, each Contributor hereby
grants to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free,
irrevocable (except as stated in this section) patent license to make, have
made, use, offer to sell, sell, import, and otherwise transfer the Work, where
such license applies only to those patent claims licensable by such Contributor
that are necessarily infringed by their Contribution(s) alone or by combination
of their Contribution(s) with the Work to which such Contribution(s) was
submitted. If You institute patent litigation against any entity (including a
cross-claim or counterclaim in a lawsuit) alleging that the Work or a
Contribution incorporated within the Work constitutes direct or contributory
patent infringement, then any patent licenses granted to You under this License
for that Work shall terminate as of the date such litigation is filed.
4. Redistribution.
You may reproduce and distribute copies of the Work or Derivative Works thereof
in any medium, with or without modifications, and in Source or Object form,
provided that You meet the following conditions:
You must give any other recipients of the Work or Derivative Works a copy of
this License; and
You must cause any modified files to carry prominent notices stating that You
changed the files; and
You must retain, in the Source form of any Derivative Works that You distribute,
all copyright, patent, trademark, and attribution notices from the Source form
of the Work, excluding those notices that do not pertain to any part of the
Derivative Works; and
If the Work includes a "NOTICE" text file as part of its distribution, then any
Derivative Works that You distribute must include a readable copy of the
attribution notices contained within such NOTICE file, excluding those notices
that do not pertain to any part of the Derivative Works, in at least one of the
following places: within a NOTICE text file distributed as part of the
Derivative Works; within the Source form or documentation, if provided along
with the Derivative Works; or, within a display generated by the Derivative
Works, if and wherever such third-party notices normally appear. The contents of
the NOTICE file are for informational purposes only and do not modify the
License. You may add Your own attribution notices within Derivative Works that
You distribute, alongside or as an addendum to the NOTICE text from the Work,
provided that such additional attribution notices cannot be construed as
modifying the License.
You may add Your own copyright statement to Your modifications and may provide
additional or different license terms and conditions for use, reproduction, or
distribution of Your modifications, or for any such Derivative Works as a whole,
provided Your use, reproduction, and distribution of the Work otherwise complies
with the conditions stated in this License.
5. Submission of Contributions.
Unless You explicitly state otherwise, any Contribution intentionally submitted
for inclusion in the Work by You to the Licensor shall be under the terms and
conditions of this License, without any additional terms or conditions.
Notwithstanding the above, nothing herein shall supersede or modify the terms of
any separate license agreement you may have executed with Licensor regarding
such Contributions.
6. Trademarks.
This License does not grant permission to use the trade names, trademarks,
service marks, or product names of the Licensor, except as required for
reasonable and customary use in describing the origin of the Work and
reproducing the content of the NOTICE file.
7. Disclaimer of Warranty.
Unless required by applicable law or agreed to in writing, Licensor provides the
Work (and each Contributor provides its Contributions) on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied,
including, without limitation, any warranties or conditions of TITLE,
NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A PARTICULAR PURPOSE. You are
solely responsible for determining the appropriateness of using or
redistributing the Work and assume any risks associated with Your exercise of
permissions under this License.
8. Limitation of Liability.
In no event and under no legal theory, whether in tort (including negligence),
contract, or otherwise, unless required by applicable law (such as deliberate
and grossly negligent acts) or agreed to in writing, shall any Contributor be
liable to You for damages, including any direct, indirect, special, incidental,
or consequential damages of any character arising as a result of this License or
out of the use or inability to use the Work (including but not limited to
damages for loss of goodwill, work stoppage, computer failure or malfunction, or
any and all other commercial damages or losses), even if such Contributor has
been advised of the possibility of such damages.
9. Accepting Warranty or Additional Liability.
While redistributing the Work or Derivative Works thereof, You may choose to
offer, and charge a fee for, acceptance of support, warranty, indemnity, or
other liability obligations and/or rights consistent with this License. However,
in accepting such obligations, You may act only on Your own behalf and on Your
sole responsibility, not on behalf of any other Contributor, and only if You
agree to indemnify, defend, and hold each Contributor harmless for any liability
incurred by, or claims asserted against, such Contributor by reason of your
accepting any such warranty or additional liability.
END OF TERMS AND CONDITIONS
APPENDIX: How to apply the Apache License to your work
To apply the Apache License to your work, attach the following boilerplate
notice, with the fields enclosed by brackets "[]" replaced with your own
identifying information. (Don't include the brackets!) The text should be
enclosed in the appropriate comment syntax for the file format. We also
recommend that a file or class name and description of purpose be included on
the same "printed page" as the copyright notice for easier identification within
third-party archives.
Copyright [yyyy] [name of copyright owner]
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.

157
modules/jieba-analysis/README.md Normal file
View File

@@ -0,0 +1,157 @@
结巴分词(java版) jieba-analysis
===============================
首先感谢jieba分词原作者[fxsjy](https://github.com/fxsjy)，没有他的无私贡献，我们也不会结识到结巴
分词. 同时也感谢jieba分词java版本的实现团队[huaban](https://github.com/huaban)，他们的努力使得Java也能直接做出效果很棒的分词。
不过由于huaban已经没有再对java版进行维护，所以我自己对项目进行了开发。除了结巴分词(java版)所保留的原项目针对搜索引擎分词的功能(cut~forindex~、cut~forsearch~)，我加入了tfidf的关键词提取功能，并且实现的效果和python的jieba版本的效果一模一样！
（以下内容在基于[jieba-java版本README.md](https://github.com/huaban/jieba-analysis])的基础上，加入了对我新加入的tfidf关键词提取模块的相关说明)
***
简介
====
支持分词模式
------------
- Search模式，用于对用户查询词分词
- Index模式，用于对索引文档分词
特性
----
- 支持多种分词模式
- 全角统一转成半角
- 用户词典功能
- conf 目录有整理的搜狗细胞词库
- 因为性能原因，最新的快照版本去除词性标注，也希望有更好的 Pull
Request 可以提供该功能。
**新特性：tfidf算法提取关键词**
```{.java}
public static void main(String[] args)
{
String content="孩子上了幼儿园 安全防拐教育要做好";
int topN=5;
TFIDFAnalyzer tfidfAnalyzer=new TFIDFAnalyzer();
List<Keyword> list=tfidfAnalyzer.analyze(content,topN);
for(Keyword word:list)
System.out.println(word.getName()+":"+word.getTfidfvalue()+",");
// 防拐:0.1992,幼儿园:0.1434,做好:0.1065,教育:0.0946,安全:0.0924
}
```
<!-- 如何获取
========
- 当前稳定版本
``` {.xml}
<dependency>
<groupId>com.huaban</groupId>
<artifactId>jieba-analysis</artifactId>
<version>1.0.2</version>
</dependency>
```
- 当前快照版本
``` {.xml}
<dependency>
<groupId>com.huaban</groupId>
<artifactId>jieba-analysis</artifactId>
<version>1.0.3-SNAPSHOT</version>
</dependency>
``` -->
如何使用
========
- Demo
``` {.java}
@Test
public void testDemo() {
JiebaSegmenter segmenter = new JiebaSegmenter();
String[] sentences =
new String[] {"这是一个伸手不见五指的黑夜。我叫孙悟空，我爱北京，我爱Python和C++。", "我不喜欢日本和服。", "雷猴回归人间。",
"工信处女干事每月经过下属科室都要亲口交代24口交换机等技术性器件的安装工作", "结果婚的和尚未结过婚的"};
for (String sentence : sentences) {
System.out.println(segmenter.process(sentence, SegMode.INDEX).toString());
}
}
```
算法(wiki补充...)
=================
- \[ \] 基于 `trie` 树结构实现高效词图扫描
- \[ \] 生成所有切词可能的有向无环图 `DAG`
- \[ \] 采用动态规划算法计算最佳切词组合
- \[ \] 基于 `HMM` 模型，采用 `Viterbi` (维特比)算法实现未登录词识别
性能评估
========
- 测试机配置
``` {.screen}
Processor 2 Intel(R) Pentium(R) CPU G620 @ 2.60GHz
Memory：8GB
分词测试时机器开了许多应用(eclipse、emacs、chrome...)，可能
会影响到测试速度
```
- *测试文本*
- 测试结果(单线程，对测试文本逐行分词，并循环调用上万次)
``` {.screen}
循环调用一万次
第一次测试结果：
time elapsed:12373, rate:2486.986533kb/s, words:917319.94/s
第二次测试结果：
time elapsed:12284, rate:2505.005241kb/s, words:923966.10/s
第三次测试结果：
time elapsed:12336, rate:2494.445880kb/s, words:920071.30/s
循环调用2万次
第一次测试结果：
time elapsed:22237, rate:2767.593144kb/s, words:1020821.12/s
第二次测试结果：
time elapsed:22435, rate:2743.167762kb/s, words:1011811.87/s
第三次测试结果：
time elapsed:22102, rate:2784.497726kb/s, words:1027056.34/s
统计结果:词典加载时间1.8s左右，分词效率每秒2Mb多，近100万词。
2 Processor Intel(R) Core(TM) i3-2100 CPU @ 3.10GHz
12G 测试效果
time elapsed:19597, rate:3140.428063kb/s, words:1158340.52/s
time elapsed:20122, rate:3058.491639kb/s, words:1128118.44/s
```
使用本库项目
============
- [analyzer-solr](https://github.com/sing1ee/analyzer-solr) @sing1ee
许可证
======
jieba(python版本)的许可证为MIT，jieba(java版本)的许可证为ApacheLicence
2.0
``` {.screen}
Copyright (C) 2013 Huaban Inc
Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.
```

59
modules/jieba-analysis/build.gradle Normal file
View File

@@ -0,0 +1,59 @@
plugins {
id 'com.android.library'
id 'kotlin-android'
}
android {
namespace = 'com.huaban.jieba'
compileSdk = project.ext.compileSdk
defaultConfig {
minSdk = project.ext.minSdk
targetSdk = project.ext.targetSdk
group = 'com.huaban'
versionName = '1.0.3-SNAPSHOT (Optimized for AutoJs6)'
description = '结巴分词工具(jieba for java)'
testInstrumentationRunner = 'androidx.test.runner.AndroidJUnitRunner'
}
compileOptions {
sourceCompatibility = JavaVersion.toVersion(project.ext.javaVersion)
targetCompatibility = JavaVersion.toVersion(project.ext.javaVersion)
}
kotlinOptions {
jvmTarget = project.ext.javaVersion
}
sourceSets {
main {
// 主 Java/Kotlin 目录
java.srcDirs = ['src/main/java']
// 将资源文件夹指定为 assets 目录
assets.srcDirs = ['src/main/assets']
}
}
}
repositories {
mavenCentral()
google()
}
dependencies {
// implementation "org.jetbrains.kotlin:kotlin-stdlib:${project.ext.kotlinVersion}"
implementation 'androidx.appcompat:appcompat:1.7.0'
implementation 'org.apache.commons:commons-lang3:3.16.0'
testImplementation 'junit:junit:4.13.2'
androidTestImplementation 'androidx.test.ext:junit:1.2.1'
androidTestImplementation 'androidx.test.espresso:espresso-core:3.6.1'
}
tasks.withType(JavaCompile).configureEach {
options.encoding = 'UTF-8'
}

BIN
modules/jieba-analysis/src/main/assets/dict-chinese-chars.db.gzip Normal file
View File

Binary file not shown.

BIN
modules/jieba-analysis/src/main/assets/dict-chinese-phrases.db.gzip Normal file
View File

Binary file not shown.

BIN
modules/jieba-analysis/src/main/assets/dict-chinese-words.db.gzip Normal file
View File

Binary file not shown.

35228
modules/jieba-analysis/src/main/assets/prob_emit.txt Normal file
View File

File diff suppressed because it is too large Load Diff

35
modules/jieba-analysis/src/main/java/com/huaban/analysis/jieba/CharacterUtil.kt Normal file
View File

@@ -0,0 +1,35 @@
package com.huaban.analysis.jieba
import java.util.regex.Pattern
object CharacterUtil {
var reSkip: Pattern = Pattern.compile("(\\d+\\.\\d+|[a-zA-Z0-9]+)")
private val connectors = charArrayOf('+', '#', '&', '.', '_', '-')
internal fun isChineseLetter(ch: Char) = ch.code in 0x4E00..0x9FA5
internal fun isEnglishLetter(ch: Char) = ch.code in 0x0041..0x005A || ch.code in 0x0061..0x007A
internal fun isDigit(ch: Char) = ch.code in 0x0030..0x0039
private fun isConnector(ch: Char) = connectors.contains(ch)
fun ccFind(ch: Char) = isChineseLetter(ch) || isEnglishLetter(ch) || isDigit(ch) || isConnector(ch)
/**
* 全角 to 半角,大写 to 小写
*
* @param input
* 输入字符
* @return 转换后的字符
*/
fun regularize(input: Char) = when (input.code) {
12288 -> 32.toChar()
in 65281..65374 -> (input.code - 65248).toChar()
in 'A'.code..'Z'.code -> input + 32
else -> input
}
}

19
modules/jieba-analysis/src/main/java/com/huaban/analysis/jieba/CharsDictionaryDatabase.kt Normal file
View File

@@ -0,0 +1,19 @@
package com.huaban.analysis.jieba
import android.content.Context
open class CharsDictionaryDatabase private constructor(context: Context) : DictionaryDatabase(context) {
override val databaseName = "dict-chinese-chars.db"
companion object {
@Volatile
private var instance: CharsDictionaryDatabase? = null
fun getInstance(context: Context): CharsDictionaryDatabase = instance ?: synchronized(this) {
instance ?: CharsDictionaryDatabase(context.applicationContext).also { instance = it }
}
}
}

250
modules/jieba-analysis/src/main/java/com/huaban/analysis/jieba/DictSegment.kt Normal file
View File

@@ -0,0 +1,250 @@
package com.huaban.analysis.jieba
import java.util.*
import java.util.concurrent.ConcurrentHashMap
/**
* 词典树分段，表示词典树的一个分枝
*/
internal class DictSegment(
// 当前节点上存储的字符
private val nodeChar: Char,
) : Comparable<DictSegment> {
private val childrenSegment: MutableMap<Char, DictSegment> by lazy { ConcurrentHashMap() }
// 数组方式存储结构
private var childrenArray: Array<DictSegment?>? = null
// 当前节点存储的 Segment 数目
// storeSize <= ARRAY_LENGTH_LIMIT, 使用数组存储
// storeSize > ARRAY_LENGTH_LIMIT, 则使用 Map 存储
private var storeSize = 0
// 当前 DictSegment 状态
// 默认 0, 1 表示从根节点到当前节点的路径表示一个词
private var nodeState = 0
// 判断是否有下一个节点
private fun hasNextNode() = this.storeSize > 0
/**
* 匹配词段
*/
@JvmOverloads
fun match(charArray: CharArray, begin: Int = 0, length: Int = charArray.size, searchHit: Hit? = null): Hit {
var niceSearchHit = searchHit
if (niceSearchHit == null) {
// 如果hit为空，新建
niceSearchHit = Hit()
// 设置hit的其实文本位置
niceSearchHit.begin = begin
} else {
// 否则要将HIT状态重置
niceSearchHit.setUnmatch()
}
// 设置hit的当前处理位置
niceSearchHit.end = begin
val keyChar = charArray[begin]
var ds: DictSegment? = null
// 引用实例变量为本地变量，避免查询时遇到更新的同步问题
val segmentArray = this.childrenArray
val segmentMap: Map<Char, DictSegment> = this.childrenSegment
// STEP1 在节点中查找keyChar对应的DictSegment
if (segmentArray != null) {
// 在数组中查找
val keySegment = DictSegment(keyChar)
val position = Arrays.binarySearch(segmentArray, 0, this.storeSize, keySegment)
if (position >= 0) {
ds = segmentArray[position]
}
} else {
// 在map中查找
ds = segmentMap[keyChar]
}
// STEP2 找到DictSegment，判断词的匹配状态，是否继续递归，还是返回结果
if (ds != null) {
if (length > 1) {
// 词未匹配完，继续往下搜索
return ds.match(charArray, begin + 1, length - 1, niceSearchHit)
} else if (length == 1) {
// 搜索最后一个char
if (ds.nodeState == 1) {
// 添加HIT状态为完全匹配
niceSearchHit.setMatch()
}
if (ds.hasNextNode()) {
// 添加HIT状态为前缀匹配
niceSearchHit.setPrefix()
// 记录当前位置的DictSegment
niceSearchHit.matchedDictSegment = ds
}
return niceSearchHit
}
}
// STEP3 没有找到DictSegment， 将HIT设置为不匹配
return niceSearchHit
}
/**
* 加载填充词典片段
*/
private fun fillSegment(charArray: CharArray) {
this.fillSegment(charArray, 0, charArray.size, 1)
}
/**
* 屏蔽词典中的一个词
*/
@Suppress("unused")
fun disableSegment(charArray: CharArray) {
this.fillSegment(charArray, 0, charArray.size, 0)
}
/**
* 加载填充词典片段
*/
@Synchronized
private fun fillSegment(charArray: CharArray, begin: Int, length: Int, enabled: Int) {
// 获取字典表中的汉字对象
val beginChar = charArray[begin]
var keyChar = charMap[beginChar]
// 字典中没有该字，则将其添加入字典
if (keyChar == null) {
charMap[beginChar] = beginChar
keyChar = beginChar
}
// 搜索当前节点的存储，查询对应keyChar的keyChar，如果没有则创建
val ds = lookforSegment(keyChar, enabled)
if (ds != null) {
// 处理keyChar对应的segment
if (length > 1) {
// 词元还没有完全加入词典树
ds.fillSegment(charArray, begin + 1, length - 1, enabled)
} else if (length == 1) {
// 已经是词元的最后一个char,设置当前节点状态为enabled，
// enabled=1表明一个完整的词，enabled=0表示从词典中屏蔽当前词
ds.nodeState = enabled
}
}
}
/**
* 查找本节点下对应的keyChar的segment *
*
* @param keyChar
* @param create
* =1如果没有找到，则创建新的segment ; =0如果没有找到，不创建，返回null
* @return
*/
private fun lookforSegment(keyChar: Char, create: Int): DictSegment? {
var ds: DictSegment? = null
if (this.storeSize <= ARRAY_LENGTH_LIMIT) {
// 获取数组容器，如果数组未创建则创建数组
val segmentArray = getChildrenArray()
// 搜寻数组
val keySegment = DictSegment(keyChar)
val position = Arrays.binarySearch(segmentArray, 0, this.storeSize, keySegment)
if (position >= 0) {
ds = segmentArray[position]
}
// 遍历数组后没有找到对应的segment
if (ds == null && create == 1) {
ds = keySegment
if (this.storeSize < ARRAY_LENGTH_LIMIT) {
// 数组容量未满，使用数组存储
segmentArray[storeSize] = ds
// segment数目+1
storeSize++
Arrays.sort(segmentArray, 0, this.storeSize)
} else {
// 数组容量已满，切换Map存储
// 获取Map容器，如果Map未创建,则创建Map
val segmentMap = getChildrenMap()
// 将数组中的segment迁移到Map中
migrate(segmentArray, segmentMap)
// 存储新的segment
segmentMap[keyChar] = ds
// segment数目+1 ， 必须在释放数组前执行storeSize++ ， 确保极端情况下，不会取到空的数组
storeSize++
// 释放当前的数组引用
this.childrenArray = null
}
}
} else {
// 获取Map容器，如果Map未创建,则创建Map
val segmentMap = getChildrenMap()
// 搜索Map
ds = segmentMap[keyChar]
if (ds == null && create == 1) {
// 构造新的segment
ds = DictSegment(keyChar)
segmentMap[keyChar] = ds
// 当前节点存储segment数目+1
storeSize++
}
}
return ds
}
/**
* 获取数组容器 线程同步方法
*/
private fun getChildrenArray(): Array<DictSegment?> {
if (this.childrenArray == null) {
synchronized(this) {
if (this.childrenArray == null) {
this.childrenArray = arrayOfNulls(ARRAY_LENGTH_LIMIT)
}
}
}
return childrenArray!!
}
private fun getChildrenMap(): MutableMap<Char, DictSegment> {
return childrenSegment
}
/**
* 将数组中的segment迁移到Map中以支持高性能存储
*/
private fun migrate(segmentArray: Array<DictSegment?>, segmentMap: MutableMap<Char, DictSegment>) {
for (segment in segmentArray) {
if (segment != null) {
segmentMap[segment.nodeChar] = segment
}
}
}
/**
* 实现Comparable接口
*/
override fun compareTo(other: DictSegment): Int {
// 对当前节点存储的char进行比较
return nodeChar.compareTo(other.nodeChar)
}
fun fillSegments(words: List<String>) {
for (word in words) {
fillSegment(word.toCharArray())
}
}
companion object {
// 公用字典表，存储汉字
private val charMap: MutableMap<Char, Char> = HashMap(16, 0.95f)
// 数组大小上限
private const val ARRAY_LENGTH_LIMIT = 3
}
}

80
modules/jieba-analysis/src/main/java/com/huaban/analysis/jieba/DictionaryDatabase.kt Normal file
View File

@@ -0,0 +1,80 @@
package com.huaban.analysis.jieba
import android.content.Context
import android.database.sqlite.SQLiteDatabase
import java.io.Closeable
import java.io.File
import java.io.FileOutputStream
import java.io.InputStream
import java.security.MessageDigest
import java.util.zip.GZIPInputStream
abstract class DictionaryDatabase internal constructor(context: Context): Closeable {
abstract val databaseName: String
private var shouldForciblyCopyDatabase = false
private val compressedDatabaseName: String
get() = "$databaseName.gzip"
private val md5Key: String
get() = "database_md5_$databaseName"
val database: SQLiteDatabase by lazy {
copyDatabaseWithCompressionAndMd5(context)
SQLiteDatabase.openDatabase(context.getDatabasePath(databaseName).absolutePath, null, SQLiteDatabase.OPEN_READONLY)
}
override fun close() = database.close()
private fun copyDatabaseWithCompressionAndMd5(context: Context) {
val dbFile = File(context.getDatabasePath(databaseName).path)
val prefs = context.getSharedPreferences(PREF_NAME, Context.MODE_PRIVATE)
val storedMd5 = prefs.getString(md5Key, null)
val currentMd5 by lazy { dbFile.md5() }
// 如果数据库文件存在，检查 MD5
if (dbFile.exists()) {
if (storedMd5 != null && storedMd5 == currentMd5) {
if (!shouldForciblyCopyDatabase) return // 文件有效，无需复制或解压
}
}
// 数据文件不存在或 MD5 不匹配，重新复制
dbFile.parentFile?.mkdirs()
context.assets.open(compressedDatabaseName).use { compressedInputStream ->
GZIPInputStream(compressedInputStream).use { gzipStream -> // 解压
FileOutputStream(dbFile).use { outputStream ->
gzipStream.copyTo(outputStream)
}
}
}
// 保存 MD5 值
prefs.edit().putString(md5Key, currentMd5).apply()
}
companion object {
private const val PREF_NAME = "dict_prefs"
}
/**
* 计算文件的 MD5
*/
private fun File.md5(): String = inputStream().use { it.md5() }
private fun InputStream.md5(): String {
val md = MessageDigest.getInstance("MD5")
val buffer = ByteArray(1024)
var read: Int
while (this.read(buffer).also { read = it } != -1) {
md.update(buffer, 0, read)
}
val digest = md.digest()
return digest.joinToString("") { "%02x".format(it) }
}
}

117
modules/jieba-analysis/src/main/java/com/huaban/analysis/jieba/Hit.java Normal file
View File

@@ -0,0 +1,117 @@
/**
*
* IK 中文分词 版本 5.0
* IK Analyzer release 5.0
*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*
* 源代码由林良益(linliangyi2005@gmail.com)提供
* 版权声明 2012，乌龙茶工作室
* provided by Linliangyi and copyright 2012 by Oolong studio
*
*/
package com.huaban.analysis.jieba;
/**
* 表示一次词典匹配的命中
*/
public class Hit {
//Hit不匹配
private static final int UNMATCH = 0x00000000;
//Hit完全匹配
private static final int MATCH = 0x00000001;
//Hit前缀匹配
private static final int PREFIX = 0x00000010;
//该HIT当前状态，默认未匹配
private int hitState = UNMATCH;
//记录词典匹配过程中，当前匹配到的词典分支节点
private DictSegment matchedDictSegment;
/*
* 词段开始位置
*/
private int begin;
/*
* 词段的结束位置
*/
private int end;
/**
* 判断是否完全匹配
*/
public boolean isMatch() {
return (this.hitState & MATCH) > 0;
}
/**
*
*/
public void setMatch() {
this.hitState = this.hitState | MATCH;
}
/**
* 判断是否是词的前缀
*/
public boolean isPrefix() {
return (this.hitState & PREFIX) > 0;
}
/**
*
*/
public void setPrefix() {
this.hitState = this.hitState | PREFIX;
}
/**
* 判断是否是不匹配
*/
public boolean isUnmatch() {
return this.hitState == UNMATCH ;
}
/**
*
*/
public void setUnmatch() {
this.hitState = UNMATCH;
}
public DictSegment getMatchedDictSegment() {
return matchedDictSegment;
}
public void setMatchedDictSegment(DictSegment matchedDictSegment) {
this.matchedDictSegment = matchedDictSegment;
}
public int getBegin() {
return begin;
}
public void setBegin(int begin) {
this.begin = begin;
}
public int getEnd() {
return end;
}
public void setEnd(int end) {
this.end = end;
}
}

211
modules/jieba-analysis/src/main/java/com/huaban/analysis/jieba/JiebaSegmenter.kt Normal file
View File

@@ -0,0 +1,211 @@
package com.huaban.analysis.jieba
import android.content.Context
import com.huaban.analysis.jieba.viterbi.FinalSeg
import java.util.*
@Suppress("LocalVariableName")
class JiebaSegmenter(context: Context) {
val dictionary by lazy { WordDictionary.getInstance(context) }
private val finalSeg by lazy { FinalSeg.getInstance(context) }
enum class SegMode { INDEX, SEARCH }
private fun createDAG(sentence: String): Map<Int, MutableList<Int>> {
val dag = HashMap<Int, MutableList<Int>>()
val trie = dictionary.trie
val chars: CharArray = sentence.toCharArray()
val N = chars.size
var i = 0
var j = 0
while (i < N) {
val hit = trie.match(chars, i, j - i + 1)
if (hit.isPrefix || hit.isMatch) {
if (hit.isMatch) {
if (!dag.containsKey(i)) {
val value: MutableList<Int> = ArrayList()
dag[i] = value
value.add(j)
} else dag[i]!!.add(j)
}
j += 1
if (j >= N) {
i += 1
j = i
}
} else {
i += 1
j = i
}
}
i = 0
while (i < N) {
if (!dag.containsKey(i)) {
val value: MutableList<Int> = ArrayList()
value.add(i)
dag[i] = value
}
++i
}
return dag
}
private fun calc(sentence: String, dag: Map<Int, MutableList<Int>>): Map<Int, Pair<Int>?> {
val N = sentence.length
val route = HashMap<Int, Pair<Int>?>()
route[N] = Pair(0, 0.0)
for (i in N - 1 downTo -1 + 1) {
var candidate: Pair<Int>? = null
for (x in dag[i]!!) {
val freq = dictionary.getFreq(sentence.substring(i, x + 1)) + route[x + 1]!!.freq
if (null == candidate) {
candidate = Pair(x, freq)
} else if (candidate.freq < freq) {
candidate.freq = freq
candidate.key = x
}
}
route[i] = candidate
}
return route
}
fun process(paragraph: String, mode: SegMode): List<SegToken> {
val tokens: MutableList<SegToken> = ArrayList()
var sb = StringBuilder()
var offset = 0
for (i in paragraph.indices) {
val ch = CharacterUtil.regularize(paragraph[i])
when {
CharacterUtil.ccFind(ch) -> sb.append(ch)
else -> {
if (sb.isNotEmpty()) {
// process
when (mode) {
SegMode.SEARCH -> {
for (word in sentenceProcess(sb.toString())) {
tokens.add(SegToken(word, offset, word.length.let { offset += it; offset }))
}
}
else -> {
for (token in sentenceProcess(sb.toString())) {
if (token.length > 2) {
var gram2: String?
var j = 0
while (j < token.length - 1) {
gram2 = token.substring(j, j + 2)
if (dictionary.containsWord(gram2)) tokens.add(SegToken(gram2, offset + j, offset + j + 2))
++j
}
}
if (token.length > 3) {
var gram3: String?
var j = 0
while (j < token.length - 2) {
gram3 = token.substring(j, j + 3)
if (dictionary.containsWord(gram3)) tokens.add(SegToken(gram3, offset + j, offset + j + 3))
++j
}
}
tokens.add(SegToken(token, offset, token.length.let { offset += it; offset }))
}
}
}
sb = StringBuilder()
offset = i
}
if (dictionary.containsWord(paragraph.substring(i, i + 1))) tokens.add(SegToken(paragraph.substring(i, i + 1), offset, ++offset))
else tokens.add(SegToken(paragraph.substring(i, i + 1), offset, ++offset))
}
}
}
if (sb.isNotEmpty()) when (mode) {
SegMode.SEARCH -> {
sentenceProcess(sb.toString()).mapTo(tokens) { token -> SegToken(token, offset, token.length.let { offset += it; offset }) }
}
else -> sentenceProcess(sb.toString()).forEach { token ->
if (token.length > 2) {
var gram2: String?
var j = 0
while (j < token.length - 1) {
gram2 = token.substring(j, j + 2)
if (dictionary.containsWord(gram2)) tokens.add(SegToken(gram2, offset + j, offset + j + 2))
++j
}
}
if (token.length > 3) {
var gram3: String?
var j = 0
while (j < token.length - 2) {
gram3 = token.substring(j, j + 3)
if (dictionary.containsWord(gram3)) tokens.add(SegToken(gram3, offset + j, offset + j + 3))
++j
}
}
tokens.add(SegToken(token, offset, token.length.let { offset += it; offset }))
}
}
return tokens
}
private fun sentenceProcess(sentence: String): List<String> {
val tokens: MutableList<String> = ArrayList()
val N = sentence.length
val dag = createDAG(sentence)
val route = calc(sentence, dag)
var x = 0
var y: Int
var buf: String
var sb = StringBuilder()
while (x < N) {
y = route[x]!!.key + 1
val lWord: String = sentence.substring(x, y)
when {
y - x == 1 -> sb.append(lWord)
else -> {
if (sb.isNotEmpty()) {
buf = sb.toString()
sb = StringBuilder()
when (buf.length) {
1 -> tokens.add(buf)
else -> when {
dictionary.containsWord(buf) -> tokens.add(buf)
else -> finalSeg.cut(buf, tokens)
}
}
}
tokens.add(lWord)
}
}
x = y
}
buf = sb.toString()
if (buf.isNotEmpty()) {
when (buf.length) {
1 -> tokens.add(buf)
else -> when {
dictionary.containsWord(buf) -> tokens.add(buf)
else -> finalSeg.cut(buf, tokens)
}
}
}
return tokens
}
fun cutSmall(hans: String, limit: Int): List<String> = when {
hans.isEmpty() || limit <= 0 -> emptyList()
else -> process(hans, SegMode.SEARCH)
.map { token -> token.word }
.flatMap { word ->
when {
word.length > limit -> word.chunked(limit)
else -> listOf(word)
}
}
}
}

24
modules/jieba-analysis/src/main/java/com/huaban/analysis/jieba/Log.java Normal file
View File

@@ -0,0 +1,24 @@
package com.huaban.analysis.jieba;
/**
* @description: enable output content to be controlled by switch
* @author: sharkdoodoo@foxmail.com
* @date: 2022/6/21
*/
public class Log {
private static final boolean LOG_ENABLE = Boolean.parseBoolean(System.getProperty("jieba.log.enable", "true"));
public static final void debug(String debugInfo) {
if (LOG_ENABLE) {
System.out.println(debugInfo);
}
}
public static final void error(String errorInfo) {
if (LOG_ENABLE) {
System.err.println(errorInfo);
}
}
}

13
modules/jieba-analysis/src/main/java/com/huaban/analysis/jieba/Node.java Normal file
View File

@@ -0,0 +1,13 @@
package com.huaban.analysis.jieba;
public class Node {
public Character value;
public Node parent;
public Node(Character value, Node parent) {
this.value = value;
this.parent = parent;
}
}

20
modules/jieba-analysis/src/main/java/com/huaban/analysis/jieba/Pair.java Normal file
View File

@@ -0,0 +1,20 @@
package com.huaban.analysis.jieba;
import org.jetbrains.annotations.NotNull;
public class Pair<K> {
public K key;
public Double freq;
public Pair(K key, double freq) {
this.key = key;
this.freq = freq;
}
@NotNull
@Override
public String toString() {
return "Candidate [key=" + key + ", freq=" + freq + "]";
}
}

20
modules/jieba-analysis/src/main/java/com/huaban/analysis/jieba/PhrasesDictionaryDatabase.kt Normal file
View File

@@ -0,0 +1,20 @@
package com.huaban.analysis.jieba
import android.content.Context
open class PhrasesDictionaryDatabase private constructor(context: Context): DictionaryDatabase(context) {
override val databaseName = "dict-chinese-phrases.db"
companion object {
@Volatile
private var instance: PhrasesDictionaryDatabase? = null
fun getInstance(context: Context): PhrasesDictionaryDatabase = instance ?: synchronized(this) {
instance ?: PhrasesDictionaryDatabase(context.applicationContext).also { instance = it }
}
}
}

22
modules/jieba-analysis/src/main/java/com/huaban/analysis/jieba/SegToken.java Normal file
View File

@@ -0,0 +1,22 @@
package com.huaban.analysis.jieba;
public class SegToken {
public String word;
public int startOffset;
public int endOffset;
public SegToken(String word, int startOffset, int endOffset) {
this.word = word;
this.startOffset = startOffset;
this.endOffset = endOffset;
}
@Override
public String toString() {
return "[" + word + ", " + startOffset + ", " + endOffset + "]";
}
}

94
modules/jieba-analysis/src/main/java/com/huaban/analysis/jieba/WordDictionary.kt Normal file
View File

@@ -0,0 +1,94 @@
package com.huaban.analysis.jieba
import android.content.Context
import android.database.sqlite.SQLiteDatabase
import kotlinx.coroutines.Deferred
import kotlinx.coroutines.Dispatchers
import kotlinx.coroutines.async
import kotlinx.coroutines.awaitAll
import kotlinx.coroutines.runBlocking
import kotlin.properties.Delegates
class WordDictionary(context: Context) {
private val db: SQLiteDatabase by lazy { WordDictionaryDatabase.getInstance(context).database }
private var minFreq by Delegates.notNull<Double>()
internal var trie = DictSegment(0.toChar())
private set
init {
db.rawQuery("SELECT value FROM metadata WHERE `key` = 'min_freq' LIMIT 1", null).use {
minFreq = if (it.moveToFirst()) it.getDouble(0) else Double.MAX_VALUE
}
// db.rawQuery("SELECT word FROM dictionary", null).use { cursor ->
// trie.fillSegments(generateSequence {
// if (cursor.moveToNext()) cursor.getString(cursor.getColumnIndexOrThrow("word")) else null
// })
// }
loadTrieConcurrently()
}
private fun loadTrieConcurrently() = runBlocking {
val chunkSize = 10000
val deferredResults = mutableListOf<Deferred<Unit>>()
db.rawQuery("SELECT word FROM dictionary", null).use { cursor ->
val words = mutableListOf<String>()
while (cursor.moveToNext()) {
val word = cursor.getString(cursor.getColumnIndexOrThrow("word"))
words.add(word)
if (words.size >= chunkSize) {
val chunk = words.toList()
words.clear()
deferredResults.add(async(Dispatchers.Default) {
trie.fillSegments(chunk)
})
}
}
// 最后一块数据
if (words.isNotEmpty()) {
val chunk = words.toList()
deferredResults.add(async(Dispatchers.Default) {
trie.fillSegments(chunk)
})
}
}
// 等待所有分块完成
deferredResults.awaitAll()
}
@Suppress("unused")
fun resetDict() {
trie = DictSegment(0.toChar())
}
fun containsWord(word: String?): Boolean {
if (word.isNullOrEmpty()) return false
db.rawQuery("SELECT 1 FROM dictionary WHERE word = ? LIMIT 1", arrayOf(word)).use {
return it.count > 0
}
}
fun getFreq(key: String?): Double {
if (!key.isNullOrEmpty()) {
db.rawQuery("SELECT normalized_freq FROM dictionary WHERE word = ? LIMIT 1", arrayOf(key)).use {
if (it.moveToFirst()) {
return it.getDouble(it.getColumnIndexOrThrow("normalized_freq"))
}
}
}
return minFreq
}
companion object {
@Volatile
private var INSTANCE: WordDictionary? = null
fun getInstance(context: Context) = INSTANCE ?: synchronized(this) {
INSTANCE ?: WordDictionary(context.applicationContext).also { INSTANCE = it }
}
}
}

20
modules/jieba-analysis/src/main/java/com/huaban/analysis/jieba/WordDictionaryDatabase.kt Normal file
View File

@@ -0,0 +1,20 @@
package com.huaban.analysis.jieba
import android.content.Context
open class WordDictionaryDatabase private constructor(context: Context): DictionaryDatabase(context) {
override val databaseName = "dict-chinese-words.db"
companion object {
@Volatile
private var instance: WordDictionaryDatabase? = null
fun getInstance(context: Context): WordDictionaryDatabase = instance ?: synchronized(this) {
instance ?: WordDictionaryDatabase(context.applicationContext).also { instance = it }
}
}
}

202
modules/jieba-analysis/src/main/java/com/huaban/analysis/jieba/viterbi/FinalSeg.kt Normal file
View File

@@ -0,0 +1,202 @@
package com.huaban.analysis.jieba.viterbi
import android.content.Context
import com.huaban.analysis.jieba.CharacterUtil
import com.huaban.analysis.jieba.Log
import com.huaban.analysis.jieba.Node
import com.huaban.analysis.jieba.Pair
import java.io.BufferedReader
import java.io.IOException
import java.io.InputStreamReader
import java.nio.charset.Charset
import java.util.*
class FinalSeg private constructor() {
private fun loadModel(context: Context) {
val s = System.currentTimeMillis()
val prevStatus = HashMap<Char, CharArray>().also { prevStatus = it }
prevStatus['B'] = charArrayOf('E', 'S')
prevStatus['M'] = charArrayOf('M', 'B')
prevStatus['S'] = charArrayOf('S', 'E')
prevStatus['E'] = charArrayOf('B', 'M')
val start = HashMap<Char, Double>().also { start = it }
start['B'] = -0.26268660809250016
start['E'] = -3.14e+100
start['M'] = -3.14e+100
start['S'] = -1.4652633398537678
val trans = HashMap<Char, Map<Char, Double>>().also { trans = it }
val transB: MutableMap<Char, Double> = HashMap()
transB['E'] = -0.510825623765990
transB['M'] = -0.916290731874155
trans['B'] = transB
val transE: MutableMap<Char, Double> = HashMap()
transE['B'] = -0.5897149736854513
transE['S'] = -0.8085250474669937
trans['E'] = transE
val transM: MutableMap<Char, Double> = HashMap()
transM['E'] = -0.33344856811948514
transM['M'] = -1.2603623820268226
trans['M'] = transM
val transS: MutableMap<Char, Double> = HashMap()
transS['B'] = -0.7211965654669841
transS['S'] = -0.6658631448798212
trans['S'] = transS
// val `is` = javaClass.getResourceAsStream(PROB_EMIT)!!
val `is` = context.assets.open(PROB_EMIT)
try {
val br = BufferedReader(InputStreamReader(`is`, Charset.forName("UTF-8")))
val emit = HashMap<Char, Map<Char, Double>>().also { emit = it }
var values: MutableMap<Char, Double>? = null
while (br.ready()) {
val line = br.readLine()
val tokens = line.split("\t".toRegex()).dropLastWhile { it.isEmpty() }.toTypedArray()
if (tokens.size == 1) {
values = HashMap()
emit[tokens[0][0]] = values
} else {
values!![tokens[0][0]] = tokens[1].toDouble()
}
}
} catch (_: IOException) {
Log.error(String.format(Locale.getDefault(), "%s: load model failure!", PROB_EMIT))
} finally {
try {
`is`.close()
} catch (_: IOException) {
Log.error(String.format(Locale.getDefault(), "%s: close failure!", PROB_EMIT))
}
}
Log.debug(
String.format(
Locale.getDefault(), "model load finished, time elapsed %d ms.",
System.currentTimeMillis() - s
)
)
}
fun cut(sentence: String, tokens: MutableList<String>) {
var chinese = StringBuilder()
var other = StringBuilder()
for (element in sentence) {
if (CharacterUtil.isChineseLetter(element)) {
if (other.isNotEmpty()) {
processOtherUnknownWords(other.toString(), tokens)
other = StringBuilder()
}
chinese.append(element)
} else {
if (chinese.isNotEmpty()) {
viterbi(chinese.toString(), tokens)
chinese = StringBuilder()
}
other.append(element)
}
}
if (chinese.isNotEmpty()) viterbi(chinese.toString(), tokens)
else {
processOtherUnknownWords(other.toString(), tokens)
}
}
private fun viterbi(sentence: String, tokens: MutableList<String>) {
val v = Vector<MutableMap<Char, Double>>()
var path: MutableMap<Char?, Node?> = HashMap()
v.add(HashMap())
for (state in states) {
var emP = emit!![state]!![sentence[0]]
if (null == emP) emP = MIN_FLOAT
v[0][state] = start!![state]!! + emP
path[state] = Node(state, null)
}
for (i in 1..<sentence.length) {
val vv: MutableMap<Char, Double> = HashMap()
v.add(vv)
val newPath: MutableMap<Char?, Node?> = HashMap()
for (y in states) {
var emp = emit!![y]!![sentence[i]]
if (emp == null) emp = MIN_FLOAT
var candidate: Pair<Char>? = null
for (y0 in prevStatus!![y]!!) {
var tranp = trans!![y0]!![y]
if (null == tranp) tranp = MIN_FLOAT
tranp += (emp + v[i - 1][y0]!!)
if (null == candidate) candidate = Pair(y0, tranp)
else if (candidate.freq <= tranp) {
candidate.freq = tranp
candidate.key = y0
}
}
vv[y] = candidate!!.freq
newPath[y] = Node(y, path[candidate.key])
}
path = newPath
}
val probE = v[sentence.length - 1]['E']!!
val probS = v[sentence.length - 1]['S']!!
val posList = Vector<Char>(sentence.length)
var win: Node?
win = if (probE < probS) path['S']
else path['E']
while (win != null) {
posList.add(win.value)
win = win.parent
}
posList.reverse()
var begin = 0
var next = 0
for (i in sentence.indices) {
val pos = posList[i]
when (pos) {
'B' -> begin = i
'E' -> {
tokens.add(sentence.substring(begin, i + 1))
next = i + 1
}
'S' -> {
tokens.add(sentence.substring(i, i + 1))
next = i + 1
}
}
}
if (next < sentence.length) tokens.add(sentence.substring(next))
}
private fun processOtherUnknownWords(other: String, tokens: MutableList<String>) {
val mat = CharacterUtil.reSkip.matcher(other)
var offset = 0
while (mat.find()) {
if (mat.start() > offset) {
tokens.add(other.substring(offset, mat.start()))
}
tokens.add(mat.group())
offset = mat.end()
}
if (offset < other.length) tokens.add(other.substring(offset))
}
companion object {
private var singleInstance: FinalSeg? = null
private const val PROB_EMIT = "prob_emit.txt"
private val states = charArrayOf('B', 'M', 'E', 'S')
private var emit: MutableMap<Char, Map<Char, Double>>? = null
private var start: MutableMap<Char, Double>? = null
private var trans: MutableMap<Char, Map<Char, Double>>? = null
private var prevStatus: MutableMap<Char, CharArray>? = null
private const val MIN_FLOAT = -3.14e100
@Synchronized
fun getInstance(context: Context): FinalSeg {
if (singleInstance == null) {
singleInstance = FinalSeg().apply { loadModel(context) }
}
return singleInstance!!
}
}
}